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PREFACE 


This  report  contains  an  evaluation  of  the  prorotype  Computerized  Training 
System  (CTS) , sponsored  by  the  Department  of  the  Army  and  the  U.S.  Army 
Training  and  Doctrine  Command  (TRADOC)  and  implemented  at  the  U.S.  Army  Signal 
Center  and  Fort  Gordon  (USASC&FG) . The  methodology  involved  a "lessons  learned" 
approach  and  includes  guidance  for  future  implementation  of  computer-based 
training  systems. 

The  study  was  performed  under  the  direction  and  supervision  of 
Dr.  Robert  J.  Seidel,  Vice  President  and  Director  of  Eastern  Division,  Human 
Resources  Research  Organization. 

The  research  was  supported  under  Contract  No.  DAAB09-77-C-0010.  The 
COTR  for  this  project  was  H.A.  Musselwhite,  Chief,  CTS  Field  Office,  who 
provided  both  support  and  substantive  assistance.  The  authors  greatly 
appreciate  Mr.  Musselwhite' s assistance  and  overall  evaluation  efforts  and 
those  of  Mr.  Frank  E.  Giunti,  Mr.  Donald  A.  Kimberlin  and  Mr.  Bryan  Altman, 
of  the  U.S.  Army  Training  Support  Center,  Ft.  Eustis,  Virginia,  who  contributed 
valuable  historical,  developmental,  and  technical  information  and  assistance 
to  the  authors  throughout  the  conduct  of  the  Project.  We  are  also  grateful 
to  Ms.  Judith  Pumphrey  of  HumRRO  for  organizing  and  summarizing  the  time  log 
data,  and  to  Mr.  Michael  Hillelsohn  who  provided  valuable  assistance  in 
summarizing  the  attitude  data. 


SUMMARY 


PURPOSE  (see  pp.  1-4)  1 

The  purpose  of  this  study  was  to  evaluate  the  prototype  Computerized 
Training  System  (CTS)  located  at  the  U.S.  Army  Signal  Center  and  Ft.  Gordon 
(USASC&FG),  and  to  provide  the  Department  of  the  Army  (DA),  U.S.  Army  Training 
and  Doctrine  Command  (TRADOC),  and  individual  training  directors  with  guidance 
on  the  training,  technical,  and  cost-effectiveness  of  computerized  training 
systems  in  support  of  self-pacing  and  management  of  instruction.  This 
guidance  is  based  on  the  data  gathered,  and  lessons  learned  from  the  prototype 
CTS  implementation  at  the  USASC&FG. 

Throughout  our  evaluation,  the  central  focus  was  on  information  of  the 
"lessons  learned"  variety.  Only  information  that  can  be  applied  to  further 
planning  and  decision-making  was  addressed.  We  felt  that  a summative  evaluation 
of  training  effectiveness  was  not  appropriate  at  this  time  because  the  CTS 
system  is  a prototype,  the  courses  were  only  recently  implemented  at  Ft.  Gordon, 
and  suitable  summative- type  student  performance  data  were  lacking.  Apparently 
a lack  of  communication  regarding  the  evaluation  implications  of  a prototype 
system  resulted  in  the  expectation  by  some  of  the  users  that  comparative  data 
analyses  would  be  performed.  While  this  would  have  been  desirable  in  an 
operational  setting,  comparative  data  analyses  cannot  be  expected  to  yield 
meaningful  results  in  a prototype  implementation  such  as  this  one.  A 
comparison  of  alternative  training  systems  on  the  basis  of  costs  and/or 
effectiveness  is  meaningful  only  if  the  systems  contain  courses  with  similar 


Page  numbers  in  parentheses  refer  to  the  relevant  sections  in  the  body  of  this 
report  where  a detailed  discussion  of  the  points  highlighted  in  this  Summary 
can  be  found. 


objectives,  content,  testing  conditions,  and  criteria.  The  self-paced  and  CTS 
versions  of  the  three  courses  selected  for  implementation  did  not  meet  all  of 
these  comparability  criteria. 

Evaluation  questions  to  be  addressed  in  this  study  were  selected  from 
the  Operational  Test  Plan  devised  by  the  CTS  project  personnel.  Highest 
priority  was  assigned  to  answering  those  questions  which  could  provide  the 
most  useful  guidance  concerning  the  application  of  future  computerized  training 
systems.  There  were  five  general  areas  of  concern  covered  in  this  evaluation 
study.  These  were: 

(1)  CTS  Course  Development 

(2)  CTS  Course  Administration  and  Operations 

(3)  Training  Effectiveness 

(4)  CTS  Costs 

(5)  Implementation  Issues 

In  addition,  evaluation  of  technical  effectiveness  was  based  upon  data 
previously  collected  and  reported  in  other  studies. 

CTS  BACKGROUND  (see  pp.  5-21) 

While  the  inception  of  the  CTS  project  can  be  traced  to  the  feasibility 
studies  at  Ft.  Monmouth  in  the  late  1960's  and  early  1970's,  the  actual  start 
of  the  CTS  project  can  be  identified  as  the  date  proposals  from  potential 
contractors  were  to  be  received.  That  date  was  16  July  1973.  At  that  time, 

the  primary  purpose  of  the  prototype  system  was  to  provide  computer-assisted 

instruction  at  the  Signal  School,  located  then  at  Ft.  Monmouth,  New  Jersey. 

The  project  personnel  were  to  be  those  already  included  on  the  staff  at  the 
Product  Manager's  Office  (PMO)  at  that  site. 
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Among  the  events  following  the  project's  Inception  which  were  most 
important  in  affecting  the  progress  and  achievements  of  CTS  were: 

• The  shift  in  implementation  site  from  Ft.  Monmouth  to 
Ft.  Gordon, 

• the  shift  in  its  primary  purpose  from  CAI  to  computer- 
managed  instruction  (CMI), 

• the  shift  in  emphasis  from  field  testing  a prototype  to 
evaluating  an  operational  system. 

METHODOLOGY  (aee  pp.  23-30) 

Technical  Effectiveness 

Our  findings  and  conclusions  were  derived  from  previous  technical  reports 
that  evaluated  the  hardware/software  components  of  the  system. 

CTS  Course  Development 

The  data  used  for  answering  the  specific  evaluation  questions  were 
gathered  from  a number  of  different  sources.  One  source  was  questionnaires 
administered  to  instructional  programmer  personnel  by  CTS.  A second  source 
was  an  extensive  structured  interview  administered  by  HumRRO  personnel  at 
Ft.  Gordon  to  augment  the  CTS  surveys.  Also,  data  were  gathered  from  the 
weekly  time  logs  kept  by  course  development  personnel  to  document  the  amount 
of  time  spent  on  CTS  instructional  development. 

CTS  Course  Ad ministration  and  Operations 

The  three  courses  selected  for  CTS  were:  Field  Radio  Repair  (MOS  31E20) , 
Teletypewriter  Equipment  Repair  (MOS  31J20),  and  Avionics  Communication 
Equipment  Repair  (MOS  35L20).  Data  were  gathered  on  instructor  activities 
and  responsibilities,  content  and  prerequisite  structure  of  each  course  and 
its  component  annexes,  and  manual  and  computer-generated  student  and  course 
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records.  HumRRO  personnel  reviewed  questionnaire  data,  conducted  interviews 
with  the  staff,  studied  documents  and  acquired  hands-on  experience  as 
"students. " 

Training  Effectiveness 

Performance  data  on  the  CTS  graduates  and  self-paced  graduates  (where 
feasible)  were  obtained  and  transmitted  to  HumRRO  by  CTS  and  USASC&FG 
personnel.  Student  records  contained  ACB  entry  t°st  scores,  education  data, 
and  the  dates  when  trainees  began  and  graduated  each  course.  Secondly,  we 
obtained  performance  data  down  to  the  Task  level,  including  progress Jon 
indices,  POI  hours,  actual  instruction  hours,  absentee  hours,  and  number  of 
failed  attempts  on  the  EPIS^  test  for  each  of  the  Tasks.  Finally,  student 
attitude  data  were  obtained  from  CTS  students  on  questionnaires  administered 
before  and  after  training. 

CTS  Costs 

CTS  cost  data,  maintained  for  Fiscal  Years  1973  through  1977,  were 
organized  and  analyzed  according  to  capital,  developmental,  and  operational 
costs  for  hardware,  software,  courseware,  and  evaluation  activities.  These 
data  included  civilian  and  military  personnel  costs,  augmented  by  information 
gathered  from  the  CTS  time  logs  related  to  faculty  development  and  staff 
training. 

Implementation  Issues 

Information  related  to  implementation  problems  and  achievements  was 
obtained  from  structured  interviews  and  side-effects  surveys.  The  latter 
instruments  provided  information  regarding  unanticipated  project  consequences 
or  outcomes  not  covered  in  the  other  questionnaires/interviews. 

^Evaluation  Planning  Information  Sheet 
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FINDINGS  AND  CONCLUSIONS 


Technical  Effectiveness  (see  pp.  32-37) 

The  CTS  hardware/software  configuration  of  six  minicomputers  with  128 
CRT  terminals  was  not  designed  for  optimal  use  as  a CMI  system.  Some  key 
instructional  management  functions  are  not  being  supported,  even  though  the 
current  hardware  is  underutilized.  However,  one  or  more  of  the  Display 
Controllers  could  be  modified  to  directly  support  CMI  functions.  Only  after 
this  is  done  could  an  analysis  of  its  CMI  utility  to  Ft.  Gordon  be  performed. 

CTS  Course  Development  (see  pp . 38-63) 

A systematic,  iterative  course  development  process  was  followed  which 
included  document  review,  subject-matter  selection  for  various  means  of 
presentation,  and  flowcharting  of  student/system  interactions.  The 
Instructional  Programmers  then  prepared,  evaluated,  and  revised  the  on-  and 
off-line  course  content  and  logic. 

On-line  materials  accounted  for  11.7%  of  the  POI  hours  contained  in  the 
three  courses.  However,  approximately  65%  of  the  55,000  instructional 
development  hours  was  devoted  to  the  preparation  of  on-line  materials.  On- 
line development  activities  consisted  of  original  authoring,  review,  debug, 
and  revision  rather  than  conversion  or  modification  of  existing  self-paced 
materials. 

The  average  number  of  course  development  hours  per  hour  of  on-line 
instruction  for  all  courses  was  175;  the  range  was  144:1  (35L)  to  197:1  (31J) . 

No  systematic  procedures  were  articulated  by  the  Instructional 
Programmers  for  identifying  specific  problem  areas  in  the  instruction  or 
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for  correcting  material  based  on  formal  feedback  sources  (e.g.,  CTS-generated 
reports  or  manual  records). 

Instructional  Programmers  reported  that  no  difficulties  were  encountered 
in  fitting  the  previously  developed  self-paced  instructional  materials  into 
the  CTS  instructional  model.  They  also  felt  that  the  CTS  system  enhanced 
the  effectiveness  and  efficiency  of  instructional  validation.  However,  the 
IP's  were  divided  on  how  difficult  it  was  to  follow  the  CTS  review/revision 
process  in  making  revisions  in  a timely  manner. 

Inputting  training  materials  on-line  was  hampered  by  a number  of  acute 
system  hardware/software  problems  in  which  stored  materials  were  lost,  slow- 
down of  the  debug/revision  process  occurred,  and  delays  were  experienced  in 
course  implementation. 

CTS  Course  Administration  and  Operation  ( see  pp.  64-89 ) 

In  general,  CTS  generated  reports  that  were  used  by  instructors  and 
training  managers  were  favorably  viewed.  However,  only  20%  of  the  IP's  knew 
about  a set  at  reports  that  could  be  used  for  course  revision  purposes.  Thus, 
these  reports  were  not  used  in  spite  of  their  availability.  On  the  other 
hand,  most,  personnel  were  familiar  with  the  reports  dealing  with  student 
scheduling  and  throughput. 

The  CTS  was  also  seen  as  effective  in  monitoring  and  routing  students 
from  Task  to  Task  through  the  course.  However,  instructor  opinion  was  divided 
on  whether  or  not  CTS  reduced  their  workload  in  the  areas  of  routing  students 
and  keeping  related  records. 

It  appeared  that  CTS  had  sufficient  resources  to  handle  the  student 
throughput  during  this  implementation,  inasmuch  as  no  significant  queueing 
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was  experienced.  Also,  If  the  computer  became  inoperative,  there  were  enough 
back-up  materials  and  instructor  time  available  to  provide  the  needed 
instruction.  (This  proved  to  be  no  problem,  as  less  than  12%  of  the  P01 
hours  were  on-line.) 

Training  Effectiveness  ( see  pp.  90-108) 

Training  effectiveness  data  indicated  that  CTS  and  self-paced  students 
progressed  through  the  courses  on  the  average  about  as  fast  as  the  P01 
demanded.  However,  there  were  some  variations.  In  the  31E  course,  the  CTS 
progression  index  (PI)  was  1.11;  the  self-paced  was  .84.  In  the  31J  course, 
the  average  PI  for  CTS  was  1.01;  self-paced  ranged  from  .98  to  1.08,  depending 
upon  the  particular  POI  in  effect  at  that  time.  In  the  35L  course,  both  the 
CTS  and  self-paced  PI  was  .96.  It  is  important  to  note  several  limitations 
on  any  conclusions  to  be  drawn  from  these  findings.  The  CTS  courses  included 
pretests  and/or  posttests  not  included  in  the  self-paced  versions.  Therefore, 
the  total  time,  by  definition,  included  more  material  in  the  CTS  version 
than  in  the  other.  Also,  POI  hours  varied  over  time.  Thus,  there  were  many 
Tasks  with  different  POI  hours  in  the  CTS  and  self-paced  versions  of  the 
courses.  Although,  theoretically,  Pi's  can  be  compared  independently  of  POI 
hours,  such  changes  might  have  reflected  substantive  content  alterations. 

The  effects  of  these  modifications  on  training  time  are  not  known  and,  thus, 
hinder  any  meaningful  comparisons  between  CTS  and  self-paced  training  time. 

The  total  failure  rate  of  CTS  students  in  the  31E  course  in  FY  1977,  and 
in  the  period  October  1977  to  March  1978,  was  about  11%.  In  the  35L  course, 
the  failure  rate  from  July  1977  to  May  1978  was  approximately  12%.  No  data 
were  available  on  the  31J  course,  as  it  was  still  being  validated. 
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Available  performance  data  indicated  that  CTS  students  passed  the  Task  test 
on  the  first  attempt  somewhat  more  frequently  than  did  the  self-paced  students. 

The  data  are  limited,  however,  and  are  only  suggestive.  Correlational  analyses 
showed  positive  relationship  between  the  number  of  times  a student  had  to  take 
the  CTS  administered  Task  EP1S  tests  and  the  time  to  complete  instruction. 

This  is  apparently  based  upon  the  required  remedial  sequences  following  Task 
EP1S  test  failure  in  the  CTS  courses. 

Student  responses  to  an  attitude  survey  revealed  more  of  a dislike  than 
a liking  for  this  particular  instructional  mode.  They  felt  the  instruction 
was  too  hard,  the  instructors  less  available  than  they  ah.,  Ld  be,  and  the 
variety  of  media  were  difficult  to  work  with.  They  were  also  influenced  by 
the  degree  of  system  down-time  and  the  shortage  of  terminals,  all  of  which 
they  perceived  as  impeding  their  learning.  Two-thirds  of  the  students  said 
they  would  not  prefer  more  instruction  delivered  by  CTS. 

CTS  Costs  (see  pp . 109-112  and  132-135) 

CTS  cost  data  indicated  that  the  total  amount  expended  during  the 
period  FY  1973  to  FY  1977  was  estimated  at  $7.5  million  (in  1977  dollars). 

As  one  of  our  study  goals,  we  were  to  perform  a predictive  cost-effectiveness 
analysis  of  CTS.  However,  insufficient  operational  cost  data  were  available 
to  make  any  valid  cost  predictions.  Also,  no  direct  comparison  between  the 
CTS  and  the  self-paced  courses  was  possible  since  the  costs  for  the  latter 
were  not  calculated.  It  is  possible  that  with  a modification  to  the  existing 
software  and  hardware  conf iguration,  the  current  CTS  system  could  handle  an 
on-board  training  load  of  6,000  to  8,000  students  at  Ft.  Gordon.  This  would 
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be  the  case  if  the  system  were  to  be  used  entirely  in  a CMI  mode.  In  order 
to  perform  a cost-effectiveness  analysis,  however,  the  necessary  additional 
courseware/software/hardware  costs  would  first  have  to  be  calculated.  A 
careful  analysis  needs  to  be  done  to  determine  the  additional  software  and/or 
hardware  necessary  to  handle  all  of  Ft.  Gordon's  self-paced  course  management 
by  CTS  and  whether  the  extra  expense  is  less  than  that  of  a comparable  manual 
operation.  Only  then  can  this  issue  be  resolved. 

implementation  Issues  (see  pp.  113-121) 

A number  of  critical  management  decisions  made  early  in  the  CTS  project 
are  related  to  its  implementation  problems.  First,  the  decision  was  made 
not  to  delay  the  project  in  spite  of  a change  in  site  and  purpose.  Secondly, 
a computer  literacy  training  program  was  not  undertaken  with  USASC&FG  personnel 
at  all  levels  after  the  change  in  site  was  accomplished.  Thirdly,  the 
establishment  of  a dual  chain  of  command  at  the  USASC&FG  resulted  in  ambiguous 
relationships  between  the  CTS  project  and  School  personnel.  Fourthly,  hardware 
and  software  instability  as  late  as  January  1977  should  have  delayed  on-line 
course  development  and  student  interactions  until  subsequent  stability  and 
reliability  were  obtained.  However,  no  delays  were  requested.  These  findings 
were  supported  by  the  results  of  the  side-effects  survey  (administered  in 
1976  and  1977)  which  yielded  the  following  negative  reactions.  Namely,  the 
premature  implementation  of  CTS  was  associated  with  the  following  problems: 

• Lack  of  clear  management. 

• Instability  of  hardware,  software  and  course  materials. 

• Lack  of  adequate  training  for  instructional  personnel 
related  to  computer-based  training. 

• Personnel  turbulence. 
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LESSONS  LEARNED/RECOMMENDATIONS  (see  pp.  134-142) 


Is  the  CTS  adequate  for  the  Army's  CMI  needs  at  Ft.  Gordon  or  any 
other  Army  installation?  This  question  is  not  answerable  based  on  the 

data  available  from  the  current  implementation.  Both  the  CSSEA  (1977)  and 
tue  COMTRAINS  (1978)  reports  suggest  that  with  some  modification  to  the 
software  and  the  hardware  links,  the  current  system  could  provide  sufficient 
capability  for  all  Cfll  needs  projected  at  Ft.  Gordon.  Whether  or  not  CTS  is 
or  would  be  cost-effective  in  this  capacity  will  depend  on  the  alternative 
systems  against  which  it  is  to  be  compared. 

The  following  statements  describe  the  lessons  learned  in  our  evaluation 
of  CTS  and  provides  guidance  to  those  who  may  be  planning  to  implement  an 
innovative  training  system.  Essentially,  these  lessons  learned  can  be 
reduced  to  two  main  points: 

(1)  There  must  be  an  unambiguous,  single  chain  of  management/authority 
vested  in  an  integrated  component  of  the  targeted  school. 

(2)  A certain  sequence  of  development  and  implementation  must  be 
followed  with  a prototype  system  in  order  for  adequate  conclusions 
to  be  drawn  regarding  training  effectiveness. 

The  remainder  of  this  section  details  the  guidance  deemed  appropriate  for  a 

prototype  implementation. 

• Insure  tha t there  is  universal  a greement  (or  understanding)  on 
project  purpose (s)  . 

Project  staff  and  user  expectations  must  be  clarified  and  documented 
by  consistent,  coherent  planning  agreements  prior  to  implementation. 
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• Make  certain  that  the  system  design  is  compatible  with  the 
project  purpose (s). 

The  hardware/software  requirements  for  a CAI  system  are  different  than 
those  for  CMI. 

• Insure  that  there  is  a serial  development  of  critical  system 
Components. 

First,  the  hardware  and  software  designs  must  be  developed  and  implemeted 
to  a point  where  reliable  and  stable  outputs  are  provided  without  any  actual 
course  materials  on  the  system.  Course  development  should  proceed  on  the 
system  only  after  the  testing  and  debugging  of  the  hardware  and  software. 

• Employ  an  evaluation  model  that  is  consistent  with  the  project 
purpose (s) . 

Determining  whether  or  not  the  evaluation  is  formative  or  summative 
will  focus  the  data  collection.  In  a prototype  implementation,  a formative 
process  is  needed,  involving  iterative  development,  debugging,  testing,  and 
revision  of  course  materials.  Only  in  an  operational  mode  is  a summative 
evaluation  appropriate. 

• Employ  a staffing  mix  consistent  with  project  needs . 

An  interdisciplinary  team  approach  such  as  that  used  in  CTS  should 
be  continued  in  the  future,  and  the  mix  should  be  appropriate  to  the  type  of 
project  that  is  implemented.  If  a comparative  evaluation  is  to  be  made, 
there  should  be  dedicated  staffing  prc  ided  to  collect  data  from  all  the 
alternative  training  systems  that  are  to  be  compared.  In  the  case  of  the 
CTS  project,  not  enough  resources  were  allocated  to  these  functions. 
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i Pemanence  of  key  personnel  is  essential  for  timely  completion 
of  a complex  project. 

It  is  particularly  critical  that  turbulence  be  held  to  a minimum 
when  attempting  to  implement  such  a complex  innovation  as  computer-based 
training  systems.  Loss  of  key  personnel,  such  as  was  the  case  with  CTS,  cannot 
be  tolerated.  In  the  future,  the  Army  would  do  well  to  provide  sufficient 
extensions  of  duty  tours  to  accommodate  completion  of  such  projects. 

Training  Site/Project  Interface:  Managing  the  Implementation 

• Literacy  and  orientation  programs  are  required  prior  to  site 
installation. 

The  users  at  the  target  site  must  understand  the  nature  of  the 
computer-based  innovation. 

• Commitment,  lines  of  control,  and  participatory  management  involving 
the  designated  user  site  must  be  established  prior  to  installation. 

A degree  of  control  commensurate  with  the  involvement  and  required 
support  from  the  site  should  be  provided  from  the  outset.  This  is  necessary 
in  order  to  prevent  the  system  from  being  perceived  as  an  outside  imposition 
on  the  School. 

• A single  chain  of  command  for  project  management  at  the  designated 
site  is  necessary. 

The  Project  Manager's  Office  should  be  located  at  the  implementation 
site  in  order  to  avoid  ambiguities  in  communication. 
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» Integrate  the  chain  of  command  (management)  within  the  trainii 
dire ctorate  of  the  user's  site . 

There  should  be  a single  individual  who  is  in  control  of  the 
project  responsible  to  the  School  commandant.  This  individual  should  also 
be  charged  with  and  given  the  authority  for  instructional  development. 

• Frequent  meetings  are  necessary  to  monitor  the  atmosphere  of 
expectation  and  understanding. 

Coordination  of  team  effort,  expression  of  problems,  adhering  to 
various  contingent  deadlines,  etc.,  require  regularly  scheduled  meetings. 
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The  purpose  of  this  study  was  to  evaluate  the  prototype  Computerized 
Training  System  (CTS)  located  at  USASC&FG  and  to  provide  the  Department  of 
the  Army  (DA),  U.S.  Army  Training  and  Doctrine  Command  (TRADOC) , and  individual 
training  directors  with  guidance  on  the  training,  technical,  and  cost- 
effectiveness  of  computerized  training  systems  in  support  of  self-pacing  and 
management  of  instruction.  This  guidance  is  based  on  the  data  gathered,  and 
lessons  learned  from  the  prototype  CTS  implementation  and  operational  test 
and  evaluation  at  the  U.S.  Army  Signal  Center  and  Ft.  Gordon  (USASC&FG). 

Throughout  our  evaluation,  the  central  focus  was  on  information  of  the 
"lessons  learned"  variety.  Only  information  that  can  be  applied  to  further 
planning  and  decision-making  was  addressed.  A summative  evaluation  of  training 
effectiveness  was  not  timely  due  to  the  fact  that  the  CTS  system  is  a prototype, 
the  courses  were  recently  implemented  at  Ft.  Gordon,  and  suitable  summative 
type  student  performance  data  were  lacking. 

An  extensive  evaluation  plan  was  included  as  part  of  the  Operational  Test 
Plan  (OTP)  [September  1975].  As  a result,  a considerable  number  of  question- 
naires and  survey  instruments  were  developed  by  CTS  personnel.  When  we  began 
our  study,  we  decided  to  select  the  most  critical  topics  for  evaluation. 

In  coordination  with  CTS  personnel  and  the  COTR,  we  assigned  priorities 
to  the  evaluation  questions  described  in  the  OTP.  Highest  priority  was  assigned 
to  those  evaluation  questions  which  could  provide  the  most  useful  guidance  to 
training  directors  with  respect  to  the  application  of  advanced  technology  to 
computerized  training  systems. 
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These  high  priority  evaluation  questions  are  listed  below  and  are  addressed 
individually  in  other  Sections  of  this  report. 

4 

1.  What  is  the  CTS  course  development  process  for  off-line  and  on-line 
materials? 

2.  How  much  time  is  required  to  prepare  the  training  and  test  materials 
contained  in  the  operational  self-paced  course  Annexes  to  CTS  instructional 
materials? 

3.  What  is  the  average  development  time  (hours)  required  for  one  POI 
hour  of  instruction  in  the  Computer  Assisted  Instruction/Computer  Managed 

\ 

Instruction  (CAI/CMI)  mode? 

4.  What  feedback  is  available  for  use  in  revising  instructional  materials  | 

and  tests?  How  has  it  been  used?  What  additional  feedback  is  necessary?  How 

would  it  he  used?  ^ 

5.  What  difficulties  have  been  encountered  in  fitting  the  previously  ^ 

developed  self-paced  instructional  materials  into  the  CTS  instructional  model? 

6.  To  what  extent  does  the  system  enable  timely  modification  (e.g.,  due 
to  POI  changes),  revision  and  validation  of  course  materials? 

7.  What  special  problems,  if  any,  were  encountered  when  entering  (inputting) 
training  materials  on-line? 

8.  How  useful  are  the  CTS-generated  reports  for  instructors  and  training 

managers?  j 

9.  Has  the  system  been  effective  in  monitoring  students  within  the 

respective  Tasks /Annexes?  ^ 

10.  Were  sufficient  resources  available  to  handle  the  student  load?  j 

11.  Is  there  an  adequate  back-up  capability  to  provide  instruction  during 

computer  down-time?  j 

12.  What  special  qualifications  are  required  by  the  instructional  support 
staff? 
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^ 13.  What  administrative  and  personnel  costs  were  incurred  to  establish 

in-service  training  programs? 

14.  What  unanticipated  side  effects  or  by-products  can  be  attributed  to 
the  implementation  of  CTS? 

15.  What  is  the  average  student  progression  index  by  Task,  and  Annex  for 
each  CTS  course?  What  is  the  standard  deviation?  How  many  graduates  were  there 
in  each  course? 

16.  What  was  the  number  and  percentage  of  students  who  failed  to  graduate 

from  each  CTS  course?  ‘ 

17.  What  are  student  attitudes  towards  CTS  courses? 

18.  What  is  the  relationship  between  student  scores  on  CTS  administered 
task  tests  and  Progression  Index  values?  What  percentage  of  students  passed 
CTS  tests  on  the  first  try? 

These  evaluation  questions  can  be  grouped  into  five  general  areas  of 
concern  in  this  evaluation  study.  These  are:  1 

• CTS  Course  Development  (Questions  1-7) 

• CTS  Course  Administration  and  Operations  (Questions  8-11) 

| 

• Training  Effectiveness  (Questions  15-18) 

• CTS  Costs  (Question  13). 

j 

• Implementation  Issues  (Questions  12  and  14)  - 

Also,  cost  data  other  than  that  required  by  Question  13  were  gathered  to 
determine  the  total  costs  of  CTS.  These  costs  were  used  to  support  the  cost- 
effectiveness  projections  that  were  part  of  our  required  effort. 

In  addition  to  the  questions  listed  above  regarding  CTS  implementation, 
we  focused  our  evaluation  efforts  on  other  implementation  issues  that  must  be 
considered  when  incorporating  innovative  instructional  systems  such  as  CTS 
into  operational  training  environments. 
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No  attempt  was  made  to  independently  evaluate  the  technical  effectiveness 


of  the  hardware/software  components  of  CTS.  Rather,  we  used  data  previously 
collected  and  reported  in  other  projects  (COMTRAINS,  1978;  Report  on  the 
Performance  Measurement  and  Analysis  of  Project  ABACUS  (U.S.A.  CSSEA,  15  April 
1977;  Hunter,  1976)  to  identify  and  discuss  the  system's  hardware/software 
components  as  they  related  to  our  major  areas  of  concern. 

The  ability  to  evaluate  a complex  system  such  as  CTS  requires  the  evaluator 
to  be  cognizant  of  the  forces  and  events  that  led  to  the  system's  status  at 
the  time  it  is  being  evaluated.  Ideally,  the  evaluator  should  have  been 
directly  involved  from  the  start  of  the  project,  determining  which  data  are  to 
be  collected,  how  they  are  to  be  collected,  etc.  In  the  case  of  CTS,  HumRRO 
staff  have  had  an  intimate  knowledge  of  the  project  from  its  inception,  but 
were  not  directly  involved  in  formative  data  collection  policies  or  procedures. 
It  was  necessary,  then,  to  reacquaint  ourselves  with  the  history  of  CTS,  its 
progress  to  date,  and  thereby  attempt  to  understand  the  context  within  which 
the  evaluation  data  were  collected.  Selected  aspects  of  this  project's  history 
will  be  presented  and  discussed  in  the  following  section. 


4 


I 

I 


B.  BACKGROUND 


I 
I 

The  history  of  Project  ABACUS,  as  it  became  known  (or  the  Army's  Computer- 
ized Training  System),  has  its  roots  in  the  feasibility  studies  conducted  by 
the  US  Army  Signal  Center  and  School  (USASCS)  at  Ft.  Monmouth,  New  Jersey.  The 
interest  in  CTS  started  in  1966,  with  the  Technical  Development  Plan  submitted 
to  the  Director,  Defense  Research  and  Engineering.  An  initial  feasibility  study 
was  conducted  in  late  1967  utilizing  a small  segment  (11.25  hrs)  of  Common  Basic 
Electronics  subject  matter.  The  follow-up  study  which  began  in  June  1968  and 
concluded  in  December  1971  was  oriented  toward  demonstrating  the  viability  of  CAI 
utilizing  a larger  segment  (102  hrs)  of  Common  Basic  Electronics  as  the  test 
subject  matter  on  the  IBM-1500  Instructional  System.  The  results  of  this  par- 
ticular study  supported  the  notion  of  a 30%  or  more  savings  in  training  time 
over  conventional  instruction;  and  it  also  indicated  a reduction  in  attrition 
possible  on  the  order  of  20  to  21%  using  CAI. 

The  Beginning:  Ft.  Monmouth  Test  Site 

As  a result  of  the  successful  experience  at  USASCS,  Ft.  Monmouth,  the  De- 
partment of  the  Army,  through  the  Management  Information  Systems  Directorate, 
became  interested  in  a potential  Army-wide  use  of  CAI.  As  a result  of  this,  a 
Task  Group  was  established  to  examine  the  possibility  of  the  use  of  CAI  in  tech- 
nical training  throughout  the  Army.  The  Task  Group  Report  prompted  the  Vice  Chief 
of  Staff  of  the  Army  to  issue  a Project  Implementation  Letter  (see  Attachment  1). 
This  led  to  the  establishment  on  1 August  1972  of  a project  for  the  design  and 
development  of  a prototype  system.  (More  information  is  provided  in  the  First 
Year  Annual  Report  of  Project  ABACUS,  1 August  1973.)  It  is  significant  to 
note  that  at  that  time  ABACUS  was  considered  to  be  a prototype  program, 

Dr.  Vincent  P.  Cieri  was  designated  as  the  Interim  Product  Manager. 
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The  nature  of  Project  ABACUS  was  stated  succinctly  in  the  first  year's 
annual  report  (1973)  as  follows: 

"The  mission  of  Project  ABACUS  is  to  design,  develop,  test 
and  evaluate  a 128-terminal  computerized  training  system 
utilizing  the  multi-mini  computer  concept." 

The  authorized  staffing  for  the  project  was  39  spaces,  both  military  and 
civilian.  In  August  1972,  a permanent  Product  Manager  was  appointed,  Colonel 
G.  B.  Howard.  And  subsequently,  a Department  of  the  Army  Steering  Advisory 
Group  (SAG)  was  established  to  provide  guidance  for  the  project. 

System  Specifications:  CAI 

During  the  early  months  of  1973  (through  March),  the  Product  Manager's  Of- 
fice wrote  performance  specifications  for  the  prototype  system.  The  specifica- 
tions for  hardware  and  software  resulted  in  a Request  for  Proposal  (RFP)  which 
was  initially  issued  on  18  April  1973.  This  RFP  encompassed  operating  system 
software,  CAI  language  design,  and  hardware  components.  The  course  develop- 
ment and  evaluation  portions  of  the  project,  as  well  as  overall  management, 
were  to  be  accomplished  within  the  Army  and  overseen  by  the  Product  Manager's 
Office.  During  the  initial  conceptualization  of  this  project,  that  is, 
through  March  1973,  the  system  was  expected  to  be  established  at  Ft.  Monmouth, 
New  Jersey.  The  purpose  of  the  system  at  that  time  was  primarily  to  provide 
computer-assisted  instruction  (CAI).  This  Is  reflected  by  both  the  RFP  and 
the  Operational  Test  Plan  (September  1975).  This  purpose  required  the  hardware/ 
software  to  support:  interactive,  on-line  instruction;  a significant  percentage 
of  graphics  authoring  and  administration;  a two-second  response  time;  and  other 
CAI-related  requirements. 

In  response  to  the  RFP  (which  included  11  amendments  between  18  April 
and  16  July  1973),  three  proposals  were  received.  The  close  out  date  for 
receipt  of  proposals  was  16  July  1973.  It  is  important  to  stress  again 
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that  those  proposals  were  directed  towards  meeting  the  original  goal  of  a 
CAI  system  as  Indicated  by  the  RFP;  the  site  was  to  be  Ft.  Monmouth;  and  the 
project  personnel  were  to  be  those  already  Included  on  t he  staff  at  the  Product 
Manager's  Office  (PMO),  Ft.  Monmouth. 

Since  its  inception  In  August.  1972  the  GTS  Project  Is  one  of  the  most  well- 
documented  of  all  computer-based  training  projects.  The  GTS  staff  has  produced 
five  annual  status  reports.  Technical  evaluation  has  been  reported  three  times 
formally  (Hunter,  1976;  Report  on  the  Performance  Measurement  and  Analysis  of 
the  Project  ABACUS,  15  April  1977;  and  G0MTRA1NS  Final  Report,  1978).  The  over- 
all evaluation  plan  had  also  been  published  early  In  the  project  (Operational 
Test  Plan,  September  1975).  In  addition  to  this  documentation,  personnel  from 
the  Communicative  Technology  Office  at  Ft.  F.ustis,  Virginia,  provided  to  us  a 
chronology  of  significant  events  for  the  GTS  project.  We  have  included  this 
chart  as  Figure  1 on  pp  8-14.  With  this  chart  Rervlng  an  the  illustrative 
framework,  we  will  highlight  some  of  these  significant  events  below. 

Shift  In  Site 

The  most  important  event  was  the  shift  In  implementation  site.  The  CTS 
project  was  a prototype  computerized  training  system  that  involved  lashing 
up  six  PDP  11/35  minicomputers  in  a novel  configuration  to  service  128 
terminals  on  a real  time  basis.  The  location  was  originally  to  be  Ft.  Monmouth 
where  the  Army  had  acquired  prior  experience  in  developing  CAI  materials  and 
implementing  a CAT.  system.  On  the  other  hand,  only  course  development  work- 
shops were  held  for  USASC&FG  personnel,  but  no  management  level  computing 
literacy  training  was  given  (Whitehouse,  1 August  1974). 

Accompanying  the  implementation  site  shift  was  a shift  in  purpose  of  the 
system  from  CAT  to  C MI.  This  change  evidenced  itself  in  attention  to  management 
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Figure  1.  Chronology  of  Significant  Events  - CTS  Project 
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Figure  1 (continued)  Chronology  of  Significant  Events  - CTS  Project 


[JANUARY  - MARCH  APRIL  - JUNE  JULY  - SEPTEMBER  OCTOBER  - DECEMBER 

CTS  Project  Field  Office  Fully  PM,  CTS  move  from  Ft.  I 

Administration  Staffed  16  People  Monmouth  to  Ft.  Eustis, 

Events  (May)  Va.  (July) 


Figure  1 (continued)  Chronology  of  Significant  Events  - CTS  Project 


1976 1JANQARY  - MARCH  APRIL  - JUNE  JULY  - SEPTEMBER  OCTOBER  - DECEMBER 

CTS  Project  16  Personnel  On  Board 

Administration  in  Field  Office  (Aug) 

Events 


Figure  1 (continued)  Chronology  of  Significant  Events  - CTS  Project 


1977 I JANOARY  - MARCH  APRIL  - JUNE  JULY  - SEPTEMBER  OCTOBER  - DECEMBER 

CTS. Project  9 Personnel  On  Board  CTS  Transferred  to 

Administration  in  Field  Office  (Aug)  TDI  (Oct) 

Events 


Figure  1 (continued)  Chronology  of  Significant  Events  - CTS  Project 
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^ functions  (scheduling,  graduation  prediction,  recordkeeping  and  reporting) 

with  interactive  on-line  instruction  be  relegated  primarily  to  pre-  and  post- 
testing. 

Three  USASC&FG  courses  were  selected  for  evaluation  of  the  prototype  CTS 
in  a school  environment.  The  courses  were  self-paced  and  were  presented  in  a 
computer-managed  instructional  mode  (CMI)  by  the  Army's  prototype  CTS.  The 
three  courses  are:  Field  Radio  Repair  (MOS  31E20);  Teletypewriter  Equipment 
Repair  (MOS  31J20) ; and  Avionics  Communication  Equipment  Repair  (MOS  35L20) . 

For  sake  of  brevity,  the  courses  will  be  referred  to  as  31E,  31J  and  35L. 

Subsequent  to  the  shift  to  Ft.  Gordon,  the  fourth  year  report  on  the  CTS 
Project  noted  that  "all  of  the  personnel  originally  assigned  by  USASIGS  to  the 
project  were  replaced  at  least  once"  (Kimberlin,  1 August  1976).  This  high- 
lighted a continuing  problem  of  personnel  turbulence.  As  a result,  there  was 
a loss  of  key  software  personnel,  and  administrative  and  management  changes  not 
originally  foreseen.  There  was  a shift  in  the  relationship  between  the  project 
structure  and  the  school  structure  when  the  site  was  shifted  from  Ft.  Monmouth 
to  the  Southeastern  Signal  School  at  Ft.  Gordon,  Georgia.  That  shift  occurred 
in  November  1973,  although  it  was  known  as  early  as  April  1973  that  there  would 
be  the  shift.  Thus,  knowledge  of  the  shift  was  available  prior  to  the  Bidders' 
Conference  on  4 May  1973.  It  is  also  significant  to  note  that  the  contract  was 
not  awarded  for  development  of  the  system  until  December  1973. 

A summary  of  the  events  which  occurred  related  to  procuring  the  system  for 
Ft.  Gordon  is  given  in  the  Third  Year  Status  Report  on  the  CTS  Project  (Kimberlin, 
1 August  1975,  p.  2).  These  events  are  as  follows: 

1.  April  1973:  RFP  issued  to  industry. 

2.  December  1973:  GTE  Sylvania  awarded  the  contract. 

3.  April  1974:  Initial  computer  system  delivered  to  Product  Manager's 

I Office. 
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4.  July  1974:  Initial  32-terminal  display  controller  delivered  to 
USASIGS. 

5.  May  1975:  Full  six-processor,  multi-mini  computer  system  delivered 
and  installed  with  16  terminals.  One  hundred  and  twelve  terminals 
placed  in  local  storage. 

6.  June  1975:  Communication  study  completed. 

7.  July  1975:  Communication  and  cabling  contract  negotiations. 

8.  July  1975:  FY76  maintenance  of  CTS  contract  negotiations. 


The  following  paragraph  from  the  same  report  elaborates  on  some  other 
changes  that  took  place  as  a result  of  the  project's  shift  to  Ft.  Gordon. 


The  project  as  initially  planned  for  implementation  at 
USACES  Ft.  Monmouth,  New  Jersey,  was  to  be  colocated  in  one 
building  with  the  three  selected  courses.  The  consolidation 
of  the  Army  Signal  School  at  Ft.  Gordon,  Georgia,  has  re- 
sulted in  the  computer  system  being  located  in  the  building 
separate  from  two  of  the  courses  selected  at  the  USASIGS  Ft. 
Gordon,  Georgia.  The  computer  system  is  located  in  Moran 
Hall,  with  terminals  located  in  Moran  Hall,  Brant  Hall  and 
Greely  Hall.  This  has  increased  terminal  communication  re- 
quirements relative  to  cabling,  and  the  electromagnetic 
environment  not  planned  in  the  original  installation.  To 
date,  a study  has  been  made  and  it  has  been  determined  that: 
subterranean  cable  construction  is  required  between  build- 
ings, since  there  is  no  existing  ductwork  available;  and, 
amplifiers  and  isolation  transformers  are  required  to  sta- 
blize  the  signal  circuits  between  the  computer  and  the 
remote  terminals.  The  results  and  recommendations  of  the 
study  have  served  as  the  basis  for  the  following  on-going 
actions: 

(a)  Modifying  of  the  original  GTE-Sylvania  contract. 

(b)  Obtaining  the  necessary  funding  required. 

(c)  Negotiating  the  contract  changes  with  GTE-Sylvania 
and  US  Army  Computer  Systems  Support  and  Evaluation 
Agency  (CSSEA) . 

(Kimberlin,  August  1975,  p.  3) 


16 


I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 


Course  Development 

Significant  events  relating  to  course  development  also  occurred  as  a 
i T.ult  of  the  change  from  Ft.  Monmouth  to  Ft.  Gordon.  Three  courses  were 
chosen  for  CTS  implementation  by  the  Commandant  of  USASIGS  (Whitehouse, 

1 August  1974).  These  courses  were  already  self-paced  and  had  achieved 
a 30-352  reduction  in  course  completion  time.  In  order  to  achieve  additional 
significant  training  time  savings,  a CMI  approach  would  have  had  to  be  tailored 
to  the  needs  of  those  specific  courses,  i.e.,  be  problem-oriented.  Also, 
there  was  a need  to  establish  a field  office  at  Ft.  Gordon,  and  this  led  to 
a further  set  of  problems  related  to  delays  in  course  development  in  this  new 
operational  setting.  It  should  be  noted  that  as  the  chronology  chart  indi- 
cates, course  development  work  at  Ft.  Gordon  started  in  February  1974,  yet 
the  system's  hardware  and  software  were  not  accepted  until  October  1976.  This 
is  a reversal  of  what  one  should  expect  in  implementing  a prototype  computer- 
based  training  system. 

Revisions  to  the  programs  of  instruction  meant  that  lesson  materials  that 
had  been  prepared  earlier  had  to  be  revised  (Kimberlin,  1 August  1975).  Other 
related  problems  in  lesson  material  preparation  occurred  with  the  loss  of  time 
in  site  preparation  for  the  128-terminal  system  at  Ft.  Gordon;  revision  of  the 
data  file  structure;  and  the  need  to  have  reliability" of  the  system  improved 
at  Ft.  Gordon  after  it  was  moved  from  the  contractor's  facility.  It  should  be 
noted,  however,  that  in  spite  of  all  the  delays  caused  by  the  site  shift,  no 
request  for  a delay  in  project  completion  was  made  to  higher  headquarters. 
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Project/School  Relationship  \ 

< 

Another  significant  event  in  the  history  of  this  project  concerns  the  j 

4 

administrative  arrangement  between  the  CTS  field  office  staff  and  the  | 

I 

Ft.  Gordon  Signal  School  staff.  Course  teams  were  created  for  course  develop-  j 

ment  at  Ft.  Gordon  to  allow  cooperation  between  the  field  officer  personn  1 j 

and  the  School  personnel.  A Signal  School  Task  Group  was  created  for  school 
responsibilities.  A CTS  Field  Office  group  retained  local  project  management 
responsibilities.  There  resulted  from  this  arrangement  a dual-ladder  of 
responsibility  (see  Figure  2).  The  USASC&FG/CTS  Field  Office  cooperative 
relationship  occurred  through  voluntary  actions  and  at  a very  low  organizational 
element  within  the  School.  This  isolated  the  project  from  the  School  policy 
and  decision  makers  by  several  layers  of  management. 

Hardware/ Software  Evaluations 

Establishment  of  an  Acceptance  Test  Plan  under  contract  to  HumRRO  during 
the  early  part  of  1974  signifies  another  milestone  in  the  history  of  CTS.  The 
actual  acceptance  testing  began  in  July  1974,  under  a three-phased  process.  The 
Phase  I acceptance  test  was  accomplished  in  August  1974,  during  which  time  the 
hardware  and  the  operating  system  software  was  accepted.  The  Phase  II  accep- 
tance test  was  begun  in  February  1975.  The  system  at  that  time  failed  the  second 
phase  of  the  test.  It  was  re-run  in  April  1975  and  was  passed  at  that  time. 

Phase  III  acceptance  testing  was  begun  in  February  1976,  and  was  not  completed 

until  July  1976.  There  were  a number  of  difficulties  with  the  operating  system,  I 

its  response  time  and  in  simulating  full  loads  on  the  system.  However,  the 

system  was  fully  accepted  on  a second  run  in  October  1976. 

In  addition  to  this  acceptanct  test,  there  was  a simulation  run  by  CSSEA 
to  test  out  the  technical  effectiveness  of  the  system.  The  entire  technical 

I 

I! 
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Augment  course  TM's  under  the  Operational  Control 
of  the  TASK  Group  Leader  USASESS  except  as  mutually 


evaluation  was  not  completed  by  CSSEA  until  the  period  January  - March  1977. 
Lastly,  a study  by  Appli-Mation,  Inc.,  was  performed  to  assess  the  possibilities 
for  modifying  the  prototype  CTS  to  make  it  more  efficient.  This  study  also 
was  to  determine  whether  or  not  networking  would  be  feasible  using  the  current 
system  in  order  to  replicate  it  at  other  bases  or  to  better  handle  full  loads 
of  students  at  Ft.  Gordon.  These  evaluations  are  summarized  in  the  Results 
Section  that  describes  the  technical  effectiveness  of  CTS. 

Lessons  Le arned  Approach  to  Ev a 1 u a t ion 

A number  of  other  key  events  relate  to  the  design  and  conduct  of  an 
appropriate  evaluation.  A preliminary  evaluation  plan  prepared  in  January  1974, 
was  upgraded  to  an  Operational  Test  Plan  (OTP)  in  September  1975.  It  was 
upgraded  in  order  to  take  into  account  less  emphasis  on  tutorial  CAI  and  new 
emphasis  on  computer-managed  instruction,  and  to  take  into  account  the  shift 
from  a rather  laboratory-oriented  CAI  test  to  a predominately  operational  test 
of  CMI  (Klrnber lin,  1 August  1975,  p.  6).  The  OTP  retained,  however,  a suramative 
evaluation  focus.  It  attempted  to  address  numerous  questions  relating  to 
comparative  effectiveness  and  cost-effectiveness  analyses;  viz.,  manually 
self-paced  and  comparable  courses. 

The  last  significant  event  in  this  chronology  is  that  HumRRO  was  awarded 
a contract  to  evaluate  CTS  in  July  of  1977.  Against  this  background  of 
turbulence,  change^  in  scope,  altered  purpose,  and  ambiguities  of  management, 
we  tried  to  focus  our  evaluation  based  on  the  realities  of  the  implementation. 
Therefore,  with  no  possibilities  for  legitimate  comparative  analyses,  we  took  a 
"lessons  learned"  approach.  The  output  we  hope  to  attain  is  guidance  for  future 
implementation  of  prototype  systems. 
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The  data  collection  on  the  project  ended  in  December  1977,  yet  all  three 
courses  upon  which  the  evaluation  was  to  be  based  had  not  heen  completed.  In 
fact,  one  course,  the  31J  Teletypewriter  Equipment  Repair  Course,  was  still 
not  completely  Implemented  on  the  CTS  system  until  May  1978,  because  of 
extensive  POI  changes.  As  noted  earlier,  a prototype  implementation  such  as 
CTS  can  only  legitimately  include  a "lessons  learned"  type  of  evaluation. 

Any  comparisons  between  CTS  and  self-paced  versions  of  the  courses  can  be  made 
only  on  a preliminary  basis.  Findings  should  be  used  only  as  suggestions  for 
future  modifications  to  the  CTS  system  and  to  derive  guidelines  for  valid 
comparisons  to  be  attempted  subsequently  in  the  operational  implementation  of 
this  system. 

The  next  section — Methodology — describes  our  approach  to  gather  the 
evaluative  information  needed  for  our  study. 
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C.  METHODOLOGY 


This  section  describes  the  sources  of  data  used  in  answering  the  evaluation 
questions;  the  personnel  involved  at  Ft.  Gordon  and  Ft.  Eustis  in  collecting 
these  data;  and  the  methods  employed  in  obtaining  and  analyzing  the  data.  The 
following  paragraphs  are  organized  in  terms  of  the  areas  of  concern  in  our  eval- 
uation. 

Technical  Effectiveness 

These  findings  were  obtained  from  reports  that  evaluated  the  hardware/ 
software  components  of  the  system. 

CTS  Course  Development 

A description  of  the  process  involved  in  developing  CTS  instructional  mat- 
erials for  all  three  courses  was  obtained  from  CTS  documents  that  specify  the 
activities  that  instructional  programmer  personnel  were  to  follow  when  preparing 
course  materials.  These  were  supplemented  by  interviews  with  instructional 
programmer  personnel  at  Ft.  Gordon. 

Data  related  to  answering  specific  evaluation  questions  within  the  course 
development  area  came  from  several  different  sources.  One  such  source  was 
questionnaires  selected  from  the  large  number  administered  to  instructional 
programmer  personnel  by  CTS.  The  specific  survey  questionnaires  used  in  this 
part  of  the  evaluation  are:  (1)  The  CTS  Course  Materials  Development  Survey 
(Attachment  2),  and  (2)  the  Revised  and  Alternate  Training  Materials  Survey 
(Attachment  3) . 

A second  source  of  data  for  questions  regarding  the  course  development 
process  was  an  extensive  structured  interview  (Attachment  4)  administered  by 
HumRRO  personnel  at  Ft.  Gordon  to  augment  data  obtained  from  the  CTS  Surveys. 


The  structured  Interviews  were  administered  during  the  periods  13-17  February, 
1978  and  20-24  February,  1978,  to  23  instructional  programmer  and  management 


personnel  at  Ft.  Gordon.  The  structured  intervi ews  obtai ned  additional  inform- 
ation for  each  course  on  how  much  time  was  devoted  to  developing  on-line  and 
off-line  CTS  materials,  the  amount  of  material  developed,  and  problems  encoun- 
tered by  personnel  during  these  activities.  Tables  1 and  2 list  the  personnel 
interviewed,  their  titles,  and  the  course  administrative  component  to  which  they 
belong. 

A third  source  of  data  is  the  weekly  time  logs  kept  by  course  development 
personnel  to  document  the  amount  of  their  time  spent  in  each  of  the  components 
of  the  Instructional  development  process  for  CTS.  These  logs  for  the  period 
1974  to  1977  were  transmitted  to  HumRRO  where  the  data  on  each  individual  for 
each  course  were  summarized  and  tabulated.  A copy  of  the  instructional  develop- 
ment time  log  in  shown  in  Attachment  5. 

CTS  Course  Administration  and  Operations 

Detailed  information  of  the  organization,  content,  and  instructional  process 
in  the  CTS  versions  of  the  31E,  31J  and  35L  courses  was  gathered  in  October  and 
November  of  1977  by  HumRRO  personnel  visiting  the  Signal  School.  Interviews  with 
the  staff,  study  of  documents  and  "hands-on"  experience  as  "students"  provided 
the  data  necessary  for  thorough  documentation  of  instruction,  remediation,  and 
testing  procedures  for  both  on-line  and  off-line  activities.  In  addition,  data 
were  obtained  on  instructor  activities  and  responsibilities,  content  and  pre- 
requisite structure  of  each  course  and  its  component  annexes,  and  manual  and 
computer-generated  student  and  course  records.  The  data  were  organized  and  are 
presented  in  a narrative  description  in  the  Results  section.  Other  instruments 
used  to  answer  the  appropriate  evaluation  questions  related  to  this  area  of  con- 
cern include  the: 
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Table  1.  Instructional  Programmer  Personnel  Interviewed  during  Data  Collect- 
ion, Ft.  Eustis  and  Ft.  Gordon 


Name 

Title 

Course 

C. 

Lewis 

Instructional  Programmer/Team  Leader 

31E 

SFC  Wood 

Instructional  Programmer /Ins true tor 

31E 

SFC  Visser 

Instructional  Programmer /Ins true tor 

31E 

W. 

Whitaker 

Instructional  Programmer/Team  Leader 

31E 

*J. 

Lamb 

Instructional  Programmer/Education  Specialist 

31E/35L 

F. 

Huggins 

Instructional  Programmer /Ins true tor 

31J 

SFC  Megglnson 

Instructional  Programmer /Ins true  tor 

31J 

R. 

Bury 

Instructional  Programmer/Instructor/Team  Leader 

31J 

*SFC  Hooker 

Instructional  Programmer 

31J 

Mr 

. Singleton 

Training  Specialist 

35L 

SFC  Howard 

Instructional  Programmer/Instructor/Team  Leader 

35L 

E. 

B.  Wilkins 

Instructional  Programmer/Team  Leader 

35L 

*B. 

Altman 

Instructional  Programmer/Educational  Specialist 

31J 

*H. 

A.  Musselwhite 

Instructional  Programmer/Educational  Specialist 
(EVAL) 

*D. 

A.  Kimberlln 

Technical  Advisor/F.ducational  Specialist 

*CTS  Field  Office  Personnel 
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Table  2.  Management  Personnel  Interviewed  during  Data  Collection,  Ft.  Eustls 
and  Ft.  Gordon 


Name 

Title 

Course 

Mr.  Bush 

Chief,  Communicat lons/Flight  Line  Division 

35L 

MSGT  Thomas 

NCO/Chief  Instructor 

35L 

MSGT  Turner 

NCO/Communications/Flight  Line  Division 

35L 

Mr.  Cooper 

Training  Supervisor /Course  Supervisor 

31E 

Mr.  Dickson 

Chief,  Radio  Repair  Division 

31E 

Mr.  Moore 

Division  Chief/TTY  Repair 

31J 

LTC  Campbell 

Department  Director/Communications  Section 

31E/31J 

LTC  Westbrook 

Director,  Radio  Department 

31E 

Russ  Watkins 

Shift  Supervisor/Data  Systems  Division 

Bob  Stephens 

Chief, Data  Systems  Division 

John  White 

Training  Spec. /Data  Systems  Division 
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(1)  Operational  Reports  Survey  (Attachment  6) 

(2)  Resource  Allocation  Survey  (Attachment  7) 

(3)  Instructional  Process  Survey  (Attachment  8) 

Training  Effectiveness 

Data  on  training  effectiveness  were  represented  in  the  form  of  performance 
data  on  CTS  and  self-paced  graduates,  assembled,  keypunched  and  transmitted  to 
HumRRO.  Two  hundred  and  forty-nine  students  having  complete  records  were 
identified:  105  from  the  31E  course  (46  CTS,  59  self-paced),  54  from  the  31J 
course  (36  CTS,  18  self-paced),  and  90  from  the  35L  course  (54  CTS  and  36  self- 
paced)  . 

Student  records  contained  two  classes  of  information:  The  first  class, 
called  Background  Information,  includes  EL  and  GT  scores  from  the  ACB,  education 
level  and  the  dates  when  the  student  started  and  graduated  the  course.  The 
second  class  of  information  is  labelled  Performance  Data  at  the  Task  Level.  This 
includes  the  progression  index  (P.I.),  POI  hours,  actual  instructional  hours, 
absence  hours,  and  number  of  failed  attempts  on  the  EPIS  test  for  each  Task.  The 
format  of  the  Student  Performance  Record  is  shown  on  Table  3. 

Background  information  was  obtained  from  the  computer-stored  TREDS  history 
records.  Performance  data  were  obtained  from  manual  records  (Form  54)  kept  and 
updated  by  instructors  and  stored  by  each  department  for  anywhere  from  6 to  12 
months . 

Arrangements  were  made  to  use  the  computing  facilities  at  the  Defense  Man- 
power Data  Center  (DMDC)  of  the  Defense  Logistics  Agency  (DoD) . The  Defense  Man- 
power Data  Center  maintains  a current  version  of  the  Statistical  Packages  for  the 
Social  Sciences  (SPSS)  data  analysis  package.  Because  of  the  power  and  convenience 
of  this  package,  and  DMDC's  collocation  with  HumRRO,  a decision  was  made  to  use 
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Table  3.  Format  for  CTS  and  Self-paced  Student  Performance  Records 


Card  Column (s) 


1-3 

Student  I.D.  Number 

4 

MOS 

5-7 

EL  Score 

8-10 

GT  Score 

11-12 

Education  Level 

13-16 

Start  Date 

17-20 

Graduation  Date 

23-24 

Task  I.D.  Number 

(First  Task  in  Course) 

25 

Task  Type  (CTS  or 

Self-paced) 

26-28 

POI  Time 

29-31 

Student  Learning  Time 

32-34 

Progression  Index 

35-37 

Absence  Time 

38-39 

11  No-Goes  on  EPIS 

Test 

42-58 

Repeat  of  Columns 

23-29  (Task  2) 

61-77 

Repeat  of  Columns 

23-39  (Task  3) 

80 

Card  Number  (Five 

per  Student) 

Remaining  cards  repeat  task  data  fields  for  as  many  tasks  as  there  are  in  the 
course 

Also  repeated  are  columns  1-4  and  column  80. 
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their  facilities  rather  than  those  available  at  the  Data  Systems  Division  at 
Ft.  Gordon. 

The  punchcard  records  assembled  and  keypunched  at  Ft.  Gordon  were 
received  at  HumRRO  on  April  14,  1978.  The  card  data  were  turned  over  to  DMDC 
for  computer  storage  and  analysis  by  HumRRO  personnel  using  the  SPSS  package. 

CTS  Costs 

Cost  data,  maintained  by  CTS  for  fiscal  years  1973  through  1977  were  made 
available  to  HumRRO  for  this  study.  As  presented,  the  data  were  organized  by 
capital,  developmental  and  operational  costs  for  hardware,  software,  courseware 
and  evaluation  activities.  Personnel  costs  in  dollars  for  civilian  personnel  and 
hours  for  military  personnel  were  further  broken  down  in  terms  of  activities 
directly  in  support  of  instruction  and  those  indirectly  related.  Data  on  per- 
sonnel costs  were  further  supplemented  by  information  obtained  from  the  CTS  time 
logs  described  above. 

CTS  staff  training  cost  data  were  obtained  through  sponsor-provided  inform- 
ation on  in-house  and  contractor  supplied  training.  Data  on  these  sources  of 
training  were  obtained  where  possible  for  instructors,  instructional  programmers, 
applications  programmers,  data  entry  specialists,  and  project  management.  Copies 
of  available  documentation  on  course  content  and  length  were  obtained.  Time 
spent  in  faculty  development  and  other  related  staff  training  costs  were  obtained 
from  the  CTS  time  logs. 

Implementation  Issues 

Data  related  to  implementation  issues  and  problems  identified  by  our 
study  of  CTS  were  obtained  from  several  sources.  Data  on  special  qualifications 
for  instructional  support  staff  were  obtained  from  structured  interviews 
described  above.  Another  source  of  information  on  these  issues  was  the 


side-effect  survey  administered  to  management,  instructional  programmer  and 
instructional  personnel  employed  at  the  CTS  project.  The  side-effects  survey 
covered  unanticipated  consequences  or  outcomes  not  covered  in  the  more 
structured  series  of  surveys  discussed  earlier. 
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D.  RESULTS  AND  DISCUSSION 


Ln  this  section  of  the  report,  we  will  describe  and  discuss  our  findings. 
The  organization  of  this  section  will  reflect  the  six  maior  areas  of  concern 
in  this  study: 

• Technical  Effectiveness 

• CTS  Course  Development 

• CTS  Course  Administration  and  Operations 

• Training  Effectiveness 

• CTS  Costs 

• Implementation  Issues 
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1.  Technical  Effectiveness 


In  this  section  we  will  describe  the  CTS  hardware/software  components 
and  the  findings  and  conclusions  of  several  recent  projects  that  evaluated  these 
CTS  components. 

Three  kinds  of  formal  evaluation  activities  have  been  performed  on  the 
hardware/software  components  of  CTS.  These  include  the  following: 

1.  Acceptance  Tests.  Prior  to  formal  government  acceptance  of  the  CTS 
hardware/software  system  from  the  developing  contractor,  a series  of  tests 
were  performed  to  determine  whether  the  system  met  the  specifications  as 
described  in  the  development  contract.  An  independent  organization,  HumRRO, 
monitored  and  reported  on  these  tests  (Hunter,  1976). 

2.  Performance  Measurements.  A study  of  system  resource  utilization 
was  performed  by  the  U.S.  Army  Computer  Systems  Support  and  Evaluation  Agency 
(CSSEA),  after  the  system  was  in  operational  use  at  Ft.  Gordon  (CESSEA,  1977). 

3 . Alternative  Design  for  Computer-Managed  Instruction  (COMTRAINS) . 

An  independent  contractor  was  employed  by  the  U.S.  Army  to  determine  the 
feasibility  of  enhancing  the  CTS  hardware/software  system  to  operate  in  a 
Computer-Managed  Instruction  (CMI)  mode  (Appli-Mation,  Inc.,  1978). 

Following  a brief  description  of  the  hardware/software,  design,  the  main 
points  from  the  three  evaluations  will  be  listed. 

Hardware/Software  System  Summary 

A complete  description  of  the  CTS  system  configuration  is  contained  in  the 
series  of  manuals  and  specifications  listed  in  the  reference  section  of  this 
report. 

Figure  3 shows  schematically  the  hardware  components  of  CTS. 
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Data  Base  Storage 


There  are  six  minicomputers  operating  interdependent]./.  Four  minis. 

Display  Controllers  (DC’s)  serve  to  control  student  terminals — one  for  each 
cluster  of  32  CRT  displays  with  keyboards.  One  mini  Data  Base  Controller  (DBC) 
serves  to  control  the  data  base,  and  the  sixth  serves  as  a Systems  Controller 
(SC)  to  manage  basic  timesharing  functions  of  the  system.  The  DBC  is  connected 
to  each  DC  and  to  the  SC  using  Direct  Memory  Access  (DMA)  links.  The  SC  is 
connected  to  each  DC  via  programmed  input/output  links.  All  six  minicomputers 
are  located  in  Moran  Hall.  The  128  student  terminals,  including  eight 
instructor /monitor  terminals,  are  located  in  Moran,  Greely,  and  Brant  Halls. 

The  terminals  in  Greely  and  Brant  Halls  are  hardwired  through  patch  panels  to 
their  respective  DC's  via  buried  cables  to  these  buildings  from  the  computer 
center.  The  terminals  in  Moran  Hall  are  hardwired  in  the  building  through 
patch  panels  to  their  respective  DC's. 

Acceptance  T ests--Lessons  Learned 

The  purpose  of  the  acceptance  tests  was  to  ascertain  whether  the  system 
performed  according  to  the  standards  specified  in  the  development  contract. 

The  purpose  of  acceptance  tests  was  not^  to  determine  whether  the  system 
performed  adequately  or  appropriately  for  the  applications  and  users  at 
Ft.  Gordon.  Acceptance  tests  were  performed  by  the  CTS  office  and  monitored 
and  reported  on  by  HumRRO.  The  tests  were  conducted  in  three  phases  as  follows: 

Phase  I:  Test  of  one  Display  Controller  and  associated  terminal 
Operations 

Phase  II:  Test  of  total  system  hardware  and  software 
Phase  III:  Test  of  total  system  under  operational  conditions 

These  tests  were  conducted  over  a period  of  28  months.  Each  Phase  required 
several  retest  activities  due  to  Inability  of  the  system  to  perform  adequately 
on  its  first  attempts.  These  retesting  activities  added  to  the  government's 
cost  of  the  system,  due  to  costs  of  personnel  time,  travel,  and  additional 
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contracts  involved.  The  cost  of  monitoring  and  reporting  acceptance  tests  by 
HumRRO  was  increased  by  100%,  during  the  period  of  the  three-phased  acceptance 
test . 

One  way  of  avoiding  unnecessary  costs  involved  in  acceptance  testing  would 
have  been  to  include  in  the  original  contractor  specifications  a requirement 
that  the  contractor  perform  complete  system  tests , in-house  and  report  on  these 
tests  prior  to  the  government's  acceptance  tests.  -This  requirement  could  have 
specified  the  hardware,  software,  environment,  and  use  conditions  under  which 
tests  should  be  conducted,  as  well  as  system  performance  criteria. 

Another  major  problem  encountered  in  acceptance  testing  was  the  inability 
to  create  realistic  operational  loads  on  the  system.  The  CTS  field  office  was 
unable  to  obtain  the  large  number  of  students  that  would  be  necessary  to  test 
actual  student  load.  The  Deputy  Commandant  observed  that  the  students  were 
being  used  and  the  system  continued  to  fail.  He  ordered  that  no  students  be 
used  until  the  contractor  could  assure  the  users  a reasonable  chance  of  completing 
the  test  without  failure  of  the  system.  Although  a load  simulator  program  was 
provided  by  the  contractor,  this  simulator  did  not  include  enough  features  to 
adequately  exercise  the  system  and  create  realistic  load  conditions.  The 
system  would  operate  reliably  under  the  simulator,  but  not  when  live  users 
operated  the  system.  Therefore,  realistic  response  time  data  could  not  be 
gathered . 

A third  major  problem  was  the  lack  of  a requirement  for  the  contractor 
to  thoroughly  test  all  program  changes  and  document  them  prior  to  implementing 
them  on  the  operational  copy  of  the  software.  Lack  of  control  over  debugging, 
patching,  and  retesting  of  system  software  during  acceptance  tests  resulted  in 
confusion  as  to  the  cause  of  system  failures  and  considerable  system  down  time 
during  acceptance  tests. 
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Performance  Measurements--Lessons  Learned 


A Performance  Measurement  and  Analysis  (PMA)  was  conducted  by  the  U.S. 

Army  Computer  Systems  Support  and  Evaluation  Agency  (CSSEA,  1977)  . The  purpose 
of  the  PMA  was  to  provide  CTS  with  information  on  the  extent  to  which  various 
components  of  the  6 minicomputer  hardware  system  were  being  utilized  at 
Ft.  Gordon  with  the  current  applications,  and  workload.  The  PMA  was  conducted 
in  March  1977. 

Utilization  of  system  components  was  measured  under  simulated  conditions 
of  an  operational  load  of  400  students  using  the  three  USASC&FG  course 
applications  but  with  no  on-line  instructional  programming  activity  during  that 
time.  The  analysts  concluded  that  the  hardware  has  a high  proportion  of  unused 
capacity  under  these  conditions.  Unused  capacity  was  calculated  to  be  94.42% 
for  the  high  speed  input/output  channels  between  the  DBC  and  the  DC's. 

Although  central  processor  activity  for  the  DBC  was  not  measured  directly,  the 
low  input/output  activity  to  the  DBC  would  indicate  low  CPU  activity  as  well. 

(The  functions  of  the  DBC  require  very  little  CPU  activity  per  data  transfer.) 

CPU  idle  time  for  the  DBC  was  estimated  at  over  90%.  Further,  all  of  the 
course  materials  for  the  three  courses  at  Ft.  Gordon  were  held  on  one  of  the 
four  disk  drives  on  the  DBC.  In  sum,  CSSEA  concluded  that  CTS,  as  configured 
and  designed  under  the  envisioned  workload  at  Ft.  Gordon,  was  underutilized. 

Alternative  Design  for  Computer  Managed  Instruction 

One  of  the  reasons  that  the  present  configuration  was  underutilized  is  that 
it  was  designed  for  use  as  a tutorial  system  in  which  students  were  to  receive 
nearly  all  their  instruction  by  computer.  At  the  USASC&FG  and  at  other  Army 
schools,  however,  computer  support  to  instruction  is  currently  envisioned  in 
a management  support  role.  The  system  provides  testing  and  diagnostic  functions, 
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as  well  as  Individual  student  record  keeping  and  graduation  prediction  activities. 
Project  COMTRAINS  (Appll-Mation,  1978)  analyzed  the  CTS  configuration  and  proposed 
alternative  functions  for  the  system  as  well  as  modifications  to  the  hardware/ 
software  system. 

This  study  recommended  that  the  CTS  be  modified  to  support  additional  in- 
structional management  functions,  including  resource  allocation,  student 
scheduling,  improved  graduation  prediction,  user  feedback,  and  remedial  read- 
ing. Several  alternative  hardware/software  configurations,  taking  maximum 
advantage  of  the  existing  hardware,  were  suggested.  The  existing  PDP  11/35 
computers  can  be  used  for  CMI  controllers  by  adding  memory  and  peripheral 
equipment.  The  terminals  to  be  provided  would  be  mark-sense  readers  in  combi- 
nation with  a low  speed  printer  and  a CRT  keyboard-display  unit.  A more  advanced 
system  would  consist  of  a distributed  network  of  minicomputers  functioning  as 
CMI  controllers.  Such  computers  should  be  the  family  of  machines  of  various 
sizes  in  order  to  permit  installation  of  equipment  of  an  appropriate  size  at  each 
Army  school  or  other  training  location.  (Appli-Mbtion,  1978). 

Simmary 

The  CTS  hardware/software  configuration  of  six  minicomputers  with  128  CRT 
terminals  was  not  appropriately  designed  to  optimally  meet  requirements  for 
computer-managed  instruction  at  USASC&FG  or  at  other  Army  training  installations. 
Current  hardware  is  underutilized,  while  some  key  instructional  management 
functions  are  not  being  supported.  One  or  more  of  the  DC’s  could  be  modified 
to  support  CMI  functions  directly.  Once  such  changes  are  made,  an  analysis  of 
the  CMI  utility  of  CTS  could  be  performed. 


37 


2 . CTS  Course  Development 


In  this  section  we  will  describe  the  CTS  course  development  process  and 
discuss  the  first  seven  evaluation  questions  listed  in  Section  A of  the 
report.  The  three  courses  evaluated  were:  Field  Radio  Repair  (MOS  31F,2C)  ; 
Teletypewriter  Equipment  Repair  (MOS  31J20);  and  Avionics  Communication 
Requirement  Repair  (MOS  35L20) . 
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1.  What  is  the  CTS  course  development  process  for  off-line  and  on-line 
materials? 


' The  activities  involved  in  preparing  instructional  materials  for  either 

presentation  or  management  by  the  CTS  are  outlined  generally  in  Figure  4, 
and  in  detail  in  Figures  5 and  6. 

These  figures  show  that  the  development  of  CTS  course  materials  entailed 
the  instructional  programmer  first  reviewing  available  documentation  on  course 
or  subject-matter  requirements  such  as  existing  self-paced  performance  guides, 
task  analysis  information  sheets,  handouts  describing  the  CTS  instructional 
model,  and  detailed  flowcharts  describing  the  desired  interaction  between 
student  and  on-line  as  well  as  off-line  components  of  instruction.  Following 
this  review,  on-line  instructional  content  and  administrative  directions  were 
prepared,  followed  by  the  development  of  the  CLAAS  I language  code  which  imple- 
mented the  necessary  student  directions  and  data  recording  sub-routines,  or 
macros.  Prior  to  the  entry  of  code  and  instructional  content,  the  instruc- 
tional programmer  made  necessary  revisions  to  off-line  material  so  that  there 
was  congruence  between  these  and  the  on-line  instruction  and  directions. 

After  the  steps  of  reviewing  and  preparing  the  on-line  and  off-line  content 
and  administrative  directions  were  completed,  the  on-line  materials  were  entered 
into  the  computer  by  Instructional  Program  Entry  Specialists  (IPES) . Once  the 
* materials  were  initially  entered  into  the  system,  an  iterative  process  of  debug, 

review,  and  modification  or  revision  was  carried  out.  The  first  debug/edit 
process  occurred  after  the  material  was  entered  into  the  system  and  compiled. 
This  step  takes  care  of  any  errors  in  CLASS  I language  coding  which  violate  the 
syntax  rules  embodied  in  the  compiler.  The  second  debug/edit  review  step  in- 
volved a peer  review.  That  is,  the  instructional  content  and  directions  to  the 
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Figure  A.  Steps  In  CTS  Instructional  Development  Process 
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student,  plus  other  aspects  of  tne  course  segment  were  reviewed  and  critiqued 
by  the  instructional  programmer  and  his  colleagues,  by  instructors,  and  by  his 
team  leader.  Appropriate  changes  were  then  made. 

Following  the  peer  review,  the  instructional  segment  was  administered  to 
a small  group  of  advanced  students  who  took  the  instruction  and  answer  test 
questions  in  order  to  identify  any  components,  either  instructional  or  admin- 
istrative directions,  which  were  inaccurate,  incomplete  or  misleading.  Sub- 
sequent to  their  review,  appropriate  changes  were  made  to  on-and  off-line  text, 
directions,  and  language  code  as  necessary.  Prior  to  acceptance  and  implemen- 
tation of  the  instructional  unit  into  the  on-going  mainstream  of  the  course,  the 
material  was  administered  to  regular  students  in  the  first  full  class  to  en- 
counter the  new  material  in  order  to  identify  and  correct  any  deficiencies  or 
weaknesses  in  the  instruction  that  may  not  have  been  apparent  to  the  instructional 
programmers,  peers,  or  to  advanced  students. 
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2.  Hod  much  time  is  required  to  prepare  the  training  and  teat  materials 
contained  in  the  operational  self-paced  course  Annexes  to  CTS 
instructional  materials? 


Table  4 is  a summary  of  the  CTS  course  development  time  as  derived  from 
the  Time  Logs.  This  information  is  based  on  actual  time  charges  made  by  over 
30  instructional  programmer  personnel  during  the  3+  years  for  which  we  had 
data.  Table  3 contains  a summary  of  instructional  development  time  by  type 
of  activity  as  recalled  by  a sample  of  this  instructional  staff  during  our 
recent  interviews  (1978). 

1 

It  can  be  seen  in  Table  4 that  almost  65%  of  the  approximately  55,000  hours 
of  course  development  time  was  attributed  in  the  time  logs  to  developing  on-line 
materials. ^ The  recollections  of  our  interviewees  of  these  percentages  were 
slightly  discrepant  in  the  31J  and  35L  samplep,  and  quite  a bit  off  in  the  31E 
sample.  However,  considering  the  time  that  had  passed,  their  estimates  agreed 
substantially  with  recorded  times.  The  total  time  spent  authoring  materials  was 
estimated  by  the  13  interviewees  as  approximately  39,000  hours.  The  actual  recorded 
course  development  times  of  our  sample  totaled  approximately  31,000  hours. 

Although  our  sample  was  less  than  half  of  the  CTS  course  development  staff, 
they  accounted  for  more  than  55%  of  the  course  materials.  In  sura,  approximately 
65%  of  CTS  course  development  time  was  used  to  develop  the  on-line  materials 
(which  accounted  for  about  11.7%  of  the  POI  hours).  However,  most  of  the 
off-line  materials  had  been  previously  prepared  in  the  self-paced  version  of 
the  course  and  thus  these  percentages  should  not  be  used  to  represent  the 
relative  efficiency  or  inefficiency  of  CTS  course  development.  Rather,  these 
numbers  were  used  only  to  obtain  estimates  of  on-line  material  development  time. 

^The  time  log  data  that  we  used  did  not  include  the  time  that  Instructional 
Program  Entry  Specialists  (IPES)  spent  in  inputting  CTS  materials.  We  have  since 
learned  that  3 IPES's  were  phased  in  during  the  project  and  employed  full  time 
for  this  effort.  Such  times,  if  known,  should  be  added  to  CTS  course  develop- 
ment. However,  such  data  were  unavailable. 
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Table  4.  Summary  of  Course  Development  Time  by  Course  (from  Time  Log  Data) 

(Hours) 

I 


FY75 

FY76 

FY7T/77 

Totals 

31 E 

CTS  On-Line 

5,146 

4,072 

2,035 

11,252 

Other 

2,062 

1,139 

2,635 

5,836 

Total 

7,208 

5,211 

4,670 

17,088 

On-Line  as  % 

of  Total 

71.4  . 

78.  1 

43.6 

65.8 

31 J 

CTS  On-Line 

3,221 

5,581 

4,624 

] 3,426 

Other 

1,645 

3,473 

1,812 

6,930 

Total 

4,866 

9,054 

6,436 

20,356 

On-Line  as  % 

of  Total 

66.  2 

61.6 

71.8 

66.0 

35L 

CTS  On-Line 

3,290 

4,493 

3,172 

10,955 

Other 

2.699 

2,697 

1,150 

6,546 

Total 

5,989 

7,190 

4,322 

17,501 

On-Line  as  % 

of  Total 

5 4.9 

62.5 

73.  4 

62.6 

All  Courses 

CTS  On-Line 

11,657 

14,146 

9,831 

35,632 

Other 

6,406 

7,309 

5,597 

19,313 

Total 

18,063 

21,455 

15,428 

54,945 

On-Line  as  % 

of  Total 

64.  5 

65.9 

63.7 

64.9 

Table  5.  Summary  of  Interview  Data  by  Course  and  Author 


As  can  be  seen  in  Table  5,  instructional  development  time  was  mainly 
devoted  to  original  authoring,  review,  debug  and  revision.  Much  less  time, 
proportionately  was  spent  in  planning,  converting  existing  materials,  modifying 
materials  due  to  P01  changes,  or  coordination.  The  reason  that  converting 
existing  materials  does  not  take  up  more  time  than  reported  is  probably  due 
to  the  fact  that  most  of  the  on-line  materials  and  pre-  and  posttests  were  not 
part  of  the  self-paced  courses  had  to  be  created  from  scratch.  In  addition, 
since  over  100%  of  the  course  IP  personnel  were  replaced  by  August  1976,  the 
retraining  period  required  for  replacements  to  become  proficient  in  all  aspects 
of  an  IP's  activities  had  to  contribute  substantially  to  course  development 
time. 
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3.  What  is  the  average  development  time  (hours)  required  for  one  POI 
hour  of  instruction  in  the  CAI/CMI  mode? 


Table  6 shows  that  over  35,000  hours  were  attributed  to  on-line  CTS 
course  development  during  the  3+  years  for  which  we  had  data.  An  estimate 
was  obtained  from  CTS  personnel  that  the  on-line  portion  of  the  CTS  courses 
accounted  for  approximately  11.7%  of  the  POI  hours.  The  estimated  number  of 
on-line  hours  for  each  course  is  shown  In  the  second  column  of  Table  6. 

The  average  number  of  course  development  hours  per  hour  of  on-line  instruc- 
tion for  all  courses  was  175.  This  ratio  ranged  from  144:1  in  the  35L  course 
to  197:1  in  the  31J.  These  ratios  would  be  more  than  200:1  if  the  IPES  in- 
putting hours  were  to  be  included.  Nonetheless,  these  ratios  corr  spond  to  other 
CAI  course  development  data  in  which  materials  had  to  be  created  anew. 
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Table  6.  CTS  On-Line  Course  Development  Time 


Course 

POI  Hours 

On-Line 

Hours 

% Of 
POI 

On-Line  Course 
Development 
Hours 

Course  Development 
Hours  Per  On- 
Line  Hour 

31E 

691 

60 

8.7 

11,252 

188 

31J* 

543 

68 

12.5 

13,426 

197 

35L 

507 

76 

15.0 

10,955 

144 

Total 

1741 

204 

11.7 

35,632 

175 

POI  hours  are  shown  for  only  the  Tasks  converted  to  CTS. 
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4.  What  feedback  is  available  for  use  in  revising  instructional  materials 
and  tests?  How  has  it  been  used?  What  additional  feedback  is 
necessary?  How  would  it  he  used? 


Classroom  observation  and  CTS-generated  reports  are  two  principal  sources 
of  feedback  which  enable  instructional  programmers  to  perceive  a need  for 
revision,  to  identify  what  must  be  revised  (test  or  instruction),  and  determine 
the  nature  and  extent  of  the  necessary  changes. 

Classroom  observation  carried  out  during  and  after  the  validation  process 
enables  the  instructional  programmer  to  identify  content  and  procedural  diffi- 
culties as  they  are  encountered  by  individual  students.  CTS  reports  summarize 
class  progress  for  various  components  of  instruction  in  terms  of  time  spent  and 
performance  attained. 

Questions  regarding  feedback  were  asked  of  thirteen  instructional  programmers 
from  the  three  CTS  courses.  in  the  31E  and  31.1  courses,  the  principal  sources  of 
feedback  were  student  comments  in  the  classroom,  instructional  programmer  obser- 
vation of  the  classrooms  and  instructors,  and  review  of  their  materials  by  other 
instructional  programmers.  With  the  exception  of  the  31.1  course,  very  few 
authors  reported  that  they  made  consistent  use  of  either  CTS-generated  reports 
or  manually  kept  records.  When  asked  how  they  made  use  of  currently  available 
feedback,  very  few  of  the  authors  were  able  to  articulate  a systematic  procedure 
for  identifying  a specific  problem  area  in  the  instruction  or  for  correcting  the 
material  based  on  the  feedback.  Only  one  respondent  described  a specific 
process  involving  his  examination  of  task  reports  and  manual  records  to  identify 
problem  areas  in  the  instruction,  and  the  nature  of  the  problems.  The  other 
authors  merely  responded  with  phrases  such  as  "just  make  changes,"  "make  appro- 
priate changes,"  etc.  When  asked  what  additional  feedback  they  would  like  to 
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have,  only  three  authors  over  the  entire  three  courses  had  anything  to  say. 

One  author  asked  for  more  Instructional  validation  students;  another  author 
said  that  he  would  like  to  systematically  get  the  opinions  of  instructors  and 
instructional  programmers. 

Four  questions  in  the  CTS  Course  Materials  Development  Survey  dealt  with 
the  usefulness  of  item  analysis  reports  generated  quarterly  by  CTS.  These 
reports  provide  the  instructional  programmer  with  the  following:  (a)  the  number 
of  students  choosing  each  incorrect  alternative  on  multiple-choice  items,  and 
(b)  unanticipated  answers  for  constructed  response  items.  The  data  are  generated 
for  pre-  and  posttests  contained  in  each  TAIS.  Table  7 presents  the  text  of 
each  of  the  four  survey  items  and  summarizes  the  opinions  of  the  instructional 
programmers  who  responded. 

The  first  two  questions  deal  with  the  value  of  dlstractor  counts  in  iden- 
tifying weaknesses  in  tests  and  instructional  materials.  The  next  two  deal 
with  the  value  of  distractor  counts  and  unanticipated  responses  as  aids  to 
revision.  In  all  questions  but  the  last,  half  or  more  of  the  respondents  were 
neutral  on  the  value  of  distractor  counts;  slightly  fewer  than  half  were  neutral 
with  regard  to  unanticipated  answers.  Five  to  twenty-five  percent  of  the  res- 
pondents were  positive  in  their  opinions;  the  remaining  respondents  were  negative. 
The  sole  exception  to  this  trend  occurred  with  the  question  pertaining  to  un- 
anticipated responses  as  aids  to  revision.  Here,  48 % of  the  respondents  were 
positive. 

These  findings  indicate  that  for  the  most  part,  instructional  programmers 
were  relying  on  their  personal,  anecdotal  experiences  to  identify  and  correct 
deficiencies  in  CTS  Instructional  materials  and  tests.  It  was  found  that  only 
20%  or  fewer  respondents  knew  about  the  five  CTS  reports  that  provide  data 
useful  for  revision  of  materials.  Part  of  the  problem  stemmed  from  the  fact 
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that  reports  were  not  valid  at  the  time  of  the  evaluation,  due  to  software 
problems.  Thus,  they  did  not  use  the  CTS  reports  because  they  did  not  know 
of  them,  or  did  not  know  how  to  use  the  reports.  Findings  from  the  Operational 
Reports  Survey  (see  pp.  185-208)  suggest  the  former  interpretation. 


I 


52 


I 

I 

Table  7.  Instructional  Programmer  Personnel  Opinions  on  Districtor  Counts 

I and  Unanticipated  Responses  (from  CTS  Course  Materials  Development 

Survey) 


Strongly  Agree 

Agree 

Neutral 

Disagree 

Strongly  Disagree 

7.1.4  Distractor  counts  have  identified 

weaknesses  in  the  instructional 

material  per  se.  1 

3 

14 

2 

7.1.4  Pre-test  and  post-test  distractor 

counts  have  pinpointed  deficiencies 

inherent  in  the  test  questions.  1 

13 

7 1.4 

Distractor  counts  have  enabled 
instructional  programmers  to 
make  timely  revisions  to  ques- 
tions and  training  materials.  1 

1 

11 

1 

1 

7.1.4  Unanticipated  responses  are 

useful  when  revising  instruc- 
tional material.  2 

8 

9 

2 

1 
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5.  What  difficulties  have  been  encountered  in  fitting  the  previously 
developed  self-paced  instructional  materials  into  the  CTS  instruc- 
tional model? 


The  responses  across  the  courses  to  this  question  can  be  summarized  that, 
generally,  very  little  difficulty  was  experienced.  There  were  some  sources  of 
problems,  but  none  that  were  serious.  In  the  31E  course,  for  example,  one 
author  said  that  CTS  was  changed  to  suit  the  self-paced  performance  guides 
because  some  computer  equipment  was  unreliable,  but  there  were  very  few  changes 
in  the  self-paced  performance  guides  due  to  CTS.  In  that  course,  another  author 
said  that  there  was  some  concern  over  what  should  be  on-line,  what  should  be  off- 
line, and  what  branching  techniques  to  use.  In  the  31J  course,  author  comments 
concerning  some  problems  encountered  in  making  the  conversion  were  that  because 
POI  times  differed  between  self-paced  and  CTS  at  the  TAIS  level,  performance 
guides  and  lessons  had  to  be  redone;  that  was  not  difficult,  but  it  was  time 
consuming.  Another  author  in  the  31J  course  said  that  in  his  experience,  only  the 
course  outline  and  the  task  lists  were  used  from  the  old  self-paced  course,  that 
CTS  materials  were  developed  from  scratch;  the  CTS  course  was  developed  out  of 
a new  training  analysis  in  which  new  learning  sequences  were  developed  for  each 
of  the  200  objectives.  There  was  very  little  of  the  old  self-paced  course  used 
except  for  parts  of  the  performance  guides.  In  the  35L  course,  the  principal 
comment  made  by  an  Instructional  programmer  was  that  the  Directorate  of  Training 
Development  required  the  reading  level  to  be  simplified.  Another  author  said  that 
the  self-paced  performance  guide  designation  of  instructional  units  did  not 
correspond  to  the  CTS  designation  and  this  had  to  be  changed.  Despite  the 
individual  problems  found  by  the  various  authors,  the  overall  feeling  of  the 
instructional  programmers  was  that  what  conversion  there  was  of  self-paced  to 
CTS  did  not  present  a great  problem  in  its  execution. 
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6.  To  what  extent  does  the  system  enable  timely  modification  (e.g., 
due  to  POI  changes),  revision  and  validation  of  course  materials? 

Answers  to  these  questions  are  suggested  in  CTS  personnel  responses  to  the 
CTS  Revised  and  Alternate  Training  Materials  Survey  (Section  I)  and  from  com- 
ments concerning  POI-based  modifications  obtained  in  HummRRO's  interviews. 

The  first  part  of  Table  8 contains  the  responses  of  instructional  program- 
mer personnel  to  items  on  materials  validation  presented  in  the  Revised  and 
Alternate  Training  Materials  Survey.  The  data  indicate  that  the  majority  of 
respondents  agreed  that  validation  procedures  under  CTS  were  not  only  workable 
but  were  more  efficient  than  those  used  for  the  self-paced  or  conventional 
materials.  For  example,  7 of  13  responses  (54%)  agreed  with  the  statement 
that  it  was  easier  to  validate  materials  on-line  while  4 (31%)  did  not  agree. 
Eight  respondents  (62%)  agreed  that  using  students  in  course  validation  had 
been  helpful  in  revision  and  did  not  impair  the  review  process. 

The  second  part  of  Table  8 summarizes  items  dealing  with  revision  of 
instructional  materials.  Here,  responses  suggest  that  instructional  program- 
mer  personnel  were  at  best  divided  on  the  contribution  of  CTS  to  the  revision 
process.  According  to  the  second  part  of  the  Table,  with  the  exception  of  tests 
and  units  of  instructions,  about  as  many  respondents  perceived  more  than  a little 
difficulty  in  making  revisions  as  those  who  did  not. 

Perhaps  the  most  critical  item  concerning  revision  asks  for  a response  to 
the  statement:  "The  CTS  review  process  has  reduced  the  time  normally  required 
for  introducing  new  or  revised  materials  into  the  classroom."  Of  the  12 
responses  to  this  item,  8 (67%)  disagreed  with  the  statement.  This  suggests 
that  the  prevailing  view  among  instructional  programmer  personnel  was  that 
CTS  did  not  enable  timely  revisions  of  course  materials. 

1 
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HumRRO's  interviews  with  instructional  programmers  addressed  the  question 


I 


of  how  much  previously  developed  instructional  materials  had  to  be  discarded 
because  of  POI  changes.  This  question  was  included  because  concern  was  evi- 
denced over  the  disruption  of  the  instructional  development  process  by  the 
lack  of  non-f inalized  programs  of  instruction  for  each  course.  According  to 
the  interviews,  very  little  instructional  material  had  to  be  discarded  because 
of  POI  changes.  Instructional  programmers  did  indicate  that  there  were  cases 
where  new  materials  had  to  be  developed  because  new  pieces  of  equipment  were 
added  in  the  POI. 
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Table  8 . Revised  and  Alternate  Training  Materials  Survey 
Computerized  Training  System  (CTS) 
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Strongly  Disagre 


7.  What  special  problems , if  any , were  encountered  when  entering 
(inputting)  training  materials  on-line? 

Information  pertinent  to  this  question  was  taken  from  the  CTS  Course 
Development  Survey  and  from  interviews  of  instructional  programmers  conducted 
at  Fort  Cordon  by  HumRPO.  Table  9.  which  summarizes  responses  to  items 
from  the  survey,  is  divided  into  four  parts:  (A)  general  items;  (B)  items 
dealing  with  Instructional  Programming  Entry  Specialists  (IPES) — clerk/typists 
trained  to  input  to  the  system  CLASS  I commands  and  text  prepared  by  instruc- 
tional programmers;  (C)  items  dealing  with  CLASS  I logic  coding;  and  (D)  system 
hardware  and  software. 

Part  A of  Table  9 summarizes  two  items  dealing  with  on-line  entry  in 
general.  It  may  be  seen  from  the  distribution  of  answers  to  these  items  that 
the  majority  of  respondents  were  willing  to  agree  that  little  difficulty  was 
encountered  when  adding  or  deleting  materials  on-line;  however,  very  few  would 
concur  with  the  assertion  that  a complete  CTS  unit  can  be  changed  overnight 

The  four  items  in  Part  B indicate  that  instructional  programmers  generally 
agreed  that  the  use  of  IPES  caused  some  problems.  Most  agreed  that  IPES 
personnel  had  to  be  supervised,  that  their  typing  errors  made  on-line  editing 
more  difficult,  and  that  a shortage  of  these  specialists  caused  delays  in 
entering  materials  to  the  system.  In  addition,  IP's  reported  in  our  interviews 
that  their  time  for  lesson  development  would  actually  have  been  reduced  had 
they  been  permitted  to  enter  their  own  material  on-line.  They  felt  considerable 
time  and  effort  were  expended  in  pre-formatting  materials  so  that  IPES  could 
readily  understand  what  was  to  be  input.  In  spite  of  this  effort,  the  IP's 
spent  considerable  additional  time  correcting  typing  errors. 

Two  items  in  Part  C dealt  with  logic  coding,  that  is  the  development  of 
CALSS  I language  commands  which  when  executed  by  the  computer  implement  the 
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instructional  strategy  for  a given  unit.  Host  respondents  agreed  with  the 
statement  that  logic  coding  has  caused  few  problems  when  entering  materials 
on-line.  On  the  other  hand,  according  to  the  second  question,  more  than  a 
few  instructional  programmers  reported  logic  coding  problems  due  to  lack  of 
training  or  due  to  the  system  software  associated  with  compiling  and  executing 
the  commands.  Presumably,  problems  of  this  sort  were  the  "relatively  few 
problems"  that  mostrespondents  agreed  were  present  in  the  item  relating  logic 
coding  to  materials  entry  on-line. 

Part  D deals  with  the  role  of  the  CTS  hardware  and  software  in  problems 
related  to  on-line  entry  of  materials.  The  first  question,  couched  in  general 
terms,  reveals  that  inputting  and  "saving"  lesson  materials  was  encumbered 
principally  by  slow  system  response  time  and  the  "loss"  by  the  system  of  mat- 
erials already  entered,  requiring  re-entry  of  the  same  lessons.  A few  respond- 
ents cited  insufficient  disk  space.  This  refers  to  the  necessity  of  waiting  a 
portion  of  all  of  a day  for  disk  space  to  become  available  so  that  instructional 
programmers  could  work  on  their  materials.  The  principal  effect  of  system  down- 
time, according  to  the  second  item,  was  to  delay  course  implementation.  Nine 
of  the  14  respondents  (64%)  cited  this  as  a consequence.  The  final  question 
in  Part  D inquired  into  the  effect  of  waiting  for  system  restarts  on  preparation 
of  materials.  Ostensibly,  this  item  should  elicit  responses  similar  to  those 
given  to  the  prior  question;  however,  while  8 of  15  answers  suggest  delay,  the 
remaining  7 said  "very  little." 

Hardware /so ft ware  problems  suggested  by  the  survey  items  are  reinforced  by 
comments  made  by  instructional  programmers  during  interviews.  Instructional  pro- 
grammers were  nearly  unanimous  in  stating  that  the  inputting,  compiling  and  editing 
of  code  was  severely  hampered  by  system  instability,  very  slow  system  response 
time,  unavailability  or  limited  time  for  inputting  due  to  lack  of  disk  space 
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and  modifications  made  to  system  macros  which  required  subsequent  modification 
of  already  developed  code.  A representative  sample  of  statements  made  by 
instructional  programmers  in  all  the  courses  include:  "losing  materials  during 
crashes,"  "system  going  down,"  "some  units  wiped  out  due  to  system  problems 
(software)  and  had  to  be  re-entered,"  "compiling  problems,"  "inadequate  macros.' 
One  instructional  programmer  remarked  that  the  turnaround  time  for  e’iting  and 
inputting  was  so  slow  that  the  edit-debug-revise  process  took  at  least  3 days 
regardless  of  the  number  of  errors. 
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Table  9 . Problems  Encountered  When  Entering  Training  Materials  On-line 
(from  Course  Development  Survey) 
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A.  General 

Little  difficulty  was  experienced  when  deleting  or 
adding  new  or  revised  CTS  instructional  materials 

on-line.  0 11  3 7 1 

A complete  CTS  unit  of  instruction  can  be  changed 

over  night  eliminating  any  delay  in  student  progress.  13576 


B.  IPES 


Instructional  programmers  must  oversee  the  input  of 
course  materials  by  Instructional  Program  Entry 
Specialist  (IPES)  and/or  clerk- typists . 

Delays  were  experienced  in  entering  instructional 
materials  on-line  because  of  the  shortage  of  clerk- 
typists  and  IPES  personnel. 

Clerk-typist/IPES  personnel  experienced  little  or 
no  difficulty  in  entering  lesson  material  or  logic 
coding  into  the  system. 

Editing  materials  on-line  has  been  compounded  by 
IPES/clerk-typist  typing  errors. 


3 10  5 4 0 

3 10  8 2 0 

1 7 9 6 0 

3 10  5 3 0 


C.  Logic-Coding 

Logic-coding  of  lesson  materials  has  caused  relatively 

few  problems  when  entering  materials  into  the  system.  2 11  5 4 1 

Special  problems  encountered  when  logic-coding 
lesson  materials: 

7 Insufficient  training  on  logic-coding 
3 Changes  to  macros  created  problems 
2 None 
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Continued 


Strongly  Disagree 


Table  9 (continued).  Problems  Encountered  Uhen  Entering  Training  Materials 

On-Line  (from  Course  Development  Survey) 


D.  Hardware/ Software 

Problems  encountered  when  Inputting  and  "saving" 
lesson  material  during  the  process  of  entering 
courseware  on-line. 

6 Time  slow 

5 Lost  files  caused  instructor  to  re-enter 
materials 

3 Insufficient  disk  space 

1 None 

Adverse  effect  of  computer  down-time  on  entering 
of  materials  on  line. 

9 Delayed  course  implementation 

2 Debugging  slowed 

2 Instructional  programmer  morale  low 

1 Updating  students  slowed 

Waiting  for  system  restarts  delayed  the  preparation 

of  instructional  materials. 

7 Very  little 

6 Very  much 

2 Debugging  hindered 
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Management  Involvement  in  the  Course  Development  Process 

Supervisory  and  management  personnel  were  interviewed  within  the  same 
format  applied  to  instructional  programmers.  Questions  concerning  the  type  and 
extent  of  their  involvement  with  the  instructional  development  process  revealed 
that  supervisor  participation  in  developing  instructional  materials  was  minimal 
or  absent.  Of  the  fourteen  persons  interviewed,  ten  reported  no  participation 
in  any  aspect  of  materials  development  for  presentation  by  CTS.  One  course 
supervisor  stated  that  during  development,  about  80%  of  this  time  was  spent  in 
planning  and  developing  course  outlines  and  instructional  strategies.  Subse- 
quently, these  activities  occupied  about  10%  of  his  time.  Two  NCOICs  and  one 
course  division  chief  reported  that  between  5 and  10%  of  their  rime  was  spent 
in  reviewing  CTS  instructional  materials. 

The  apparent  remoteness  of  the  managerial  personnel  interviewed  from  CTS 
materials  development  reflects  the  organizational  structure  of  which  the  three 
CTS  courses  are  components,  and  the  multiple  responsibilities  carried  out  by 
the  interviewees.  Each  course  is  a component  of  a division  which,  in  turn., 
is  a component  of  one  of  the  several  departments  of  the  Signal  School.  Since 
the  majority  of  the  interviewees  are  in  positions  at  the  division  level  or 
higher,  their  responsibilities  extend  beyond  a specific  CTS  course.  Thus, 
while  all  respondents  indicated  managerial  activities,  including  the  acquisition, 
scheduling  and  coordination  of  resources  (money,  equipment,  staff  and  students), 
only  two  reported  more  than  5 to  15%  of  their  time  devoted  to  a CTS  course. 
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3.  CTS  Course  Administration  and  Operations 


In  this  section  we  will  describe  how  the  three  CTS  courses  operate  and 
discuss  evaluation  questions  8-11  listed  in  Section  A of  the  report. 

The  discussion  that  follows  is  organized  into  five  parts:  (1)  Organiza- 
tional Structure  of  CTS  Courses;  (2)  Description  of  CTS  Courses;  (3)  Instruc- 
tional Modes  Designated  for  CTS  Courses;  (4)  Student  Progression  Index;  and 
(5)  Generalized  CTS  Instructional  process. 

\ 

Organizational  Structure  of  CTS  Courses 

The  organizational  structure  of  the  three  CTS  courses  is  basically  the  j 

same.  The  Annex  is  the  broadest  organizational  breakdown  within  each  course. 

Each  Annex  covers  training  for  a major  equipment  type  such  as  a radio  (or  class 
of  radios^  specific  teletypewriter,  etc.  The  next  order  of  breakdown  is  the 
Task.  In  the  31J  course  there  is  one  to  one  correspondence  between  Annexes 
and  Tasks  (i.e.,  each  Annex  has  only  one  associated  Task).  On  the  other  hand, 
the  35L  course  may  have  as  many  as  five  Tasks  associated  with  an  Annex  and  the 
31E  course  as  many  as  three.  In  the  35L  course,  Tasks  could  cover  specific 
radios  within  a class  of  radios,  or  the  Task  might  cover  only  one  activity 
(such  as  align  or  troubleshoot)  for  a given  radio. 

Within  each  Task  the  smallest  topic  of  instruction  is  the  TAIS  which  de- 
r < >•  fr>ra  the  smallest  unit  of  job  activity  identified  in  Task  Analysis  In- 
• • -» i ■ r ' . - . • t . Each  TAIS  is  further  broken  down  in  the  course  into  one  or 
- • - - i • >r  an'  a performance  element.  A learning  element  presents 

• •>  *•  - • • 'in  i re. 1 for  the  TAIS  Job  component.  The  performance 

• - - if,  knowledge  to  the  behavior  (e.g., 

h.  »hle  to  perform  with 


Schematically,  the  organizational  structure  of  the  CTS  courses  is  as 


1 


i 


follows : 


Annex 


* — >Task 


i— mis 


— > Learning  Element 
— 4 Performance  Element 


At  each  breakdown  of  the  organizational  structure  students  are  administered 
on-line  written  tests  or  performance  tests.  The  test  associated  with  each 
breakdown  is  as  follows: 

Task  — A major  performance  test  (EPIS)  which  covers  all  the  behaviors 
trained  in  the  Task  breakdown  of  the  Annex.  In  the  31E  and  31J 
courses  this  performance  test  is  administered  in  a separate  test- 
ing room  by  instructors  specifically  trained  in  test  administration. 
In  the  35L  course  the  test  is  administered  in  the  instructional 
setting. 

TAIS  — This  Task  component  has  a performance  test  associated  with  it  that 
is  administered  by  the  classroom  instructor  and  covers  the  material 
trained  in  the  one  or  more  associated  learning  elements  and  the 
performance  element.  Prior  to  taking  the  TAIS  performance  test 
the  student  is  given  the  option  of  practicing  further  on  the  equip- 
ment he  or  she  is  to  be  tested  on.  When  the  student  opts  to  take 
the  test  the  instructor  (not  CTS)  determines  the  specific  test 
version  that  will  be  administered  to  the  student.  The  tests  are 


In  some  cases  these  may  be  referred  to  as  Learning/Performance  or  Practice 
Steps.  We  have  used  the  term  "element"  as  much  as  possible. 
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timed  and  the  time  is  roughly  kept  track  of  by  the  instructor. 


Quite  frequently,  the  instructor  does  not  directly  interact  with 
the  student  during  test  administration  but  rather  will  only  check 
the  student 's work  upon  completion.  Results  of  the  TAIS  test  are 
recorded  off-line  by  the  instructor  and  are  not  recorded  on  the  CTS . 

Learning  Element  — There  may  be  two  on-line  tests  associated  with  learn- 
ing elements — a pretest  and  a posttest.  There  is  always  a post- 
test covering  the  material  taught  in  the  Learning  Element.  Except 
for  the  35L  course,  where  there  are  no  pretests  employed,  pretests 
will  be  given  for  learning  elements  which  do  not  involve  the  use  of 
a job  aid.  That  is,  if  the  learning  element  involves  the  use  of 
tables  or  measuring  instruments,  or  some  sort  of  reference  in 
performance  of  the  job,  then  there  will  be  no  pretest.  Students 
have  the  option  of  either  taking  or  not  taking  the  pretests.  If 
a student  who  has  opted  to  take  the  pretest  passes  100%  of  the 
items  he  is  permitted  to  skip  the  learning  element  instruction.  If 
he  passes  less  than  100%  (the  passing  score  is  80%)  he  is  given 
what  is  called  narrative  remediation  providing  the  correct  answers 
to  the  missed  items  and  is  also  allowed  to  skip  the  learning 
element  instruction.  If  the  student  fails  the  pretest,  and  he  is 
failed  as  soon  as  he  gives  incorrect  answers  to  more  than  20%  of 
the  test  items,  he  is  routed  into  the  instruction  for  that  learning 
element.  Pre-  and  posttests  may  be  the  same  test  but  generally 
are  not.  The  posttest  is  administered  by  the  computer  when  the 
student  has  completed  the  materials  dictated  by  the  learning 
element.  A passing  score  allows  the  student  to  proceed  to  the 
performance  element  of  the  TAIS.  Failure  of  the  posttest  results 
in  directions  to  study  off-line  remedial  materials. 
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Description  of  CTS  Courses 


A brief  description  of  each  of  the  three  CTS  courses  follows: 

31E  Course. 

Students  in  the  31E  course  learn  to  maintain  4 field  radios  and  two 
radio  teletypewriter  sets.  The  course  consists  of  sixteen  Tasks  incorporated 
into  8 Annexes.  However,  students  receive  training  in  only  six  of  the  Annexes. 
Four  of  the  Annexes  (totaling  14  Tasks)  are  taught  by  CTS.  An  Annex  teaching 
maintenance  of  the  Chemical  Agent  Automatic  Alarm  (M-8)  and  an  Annex  providing 
a maintenance  shop  training  exercise  are  trained  off-line.  POI  time  allotted 
for  the  four  CTS  Annexes  of  the  31E  course  is  691  hours.  Students  enter  the 
31E  course  in  groups  every  two  weeks. 

31J  Course. 

The  31J  course  teaches  the  installation,  repair,  preventive  maintenance  and 
troubleshooting  of  a number  of  different  kinds  of  teletypewriter  machines.  There 
are  8 Annexes  (one  Task  for  each)  associated  with  the  CTS  course  and  students 
enter  in  a group  every  two  weeks.  POI  time  allotted  for  the  CTS  Annexes  of  the 
31 J course  is  543  hours. 

35L  Course. 

In  the  35L  Course  students  learn  direct  and  general  support  level  mainten- 
ance of  Army  Avionics  Communication  Equipment.  The  course  consists  of  15  Tasks 
incorporated  into  5 Annexes.  The  POI  time  allotted  for  the  CTS  Annexes  of  the 
course  is  507  hours. 


Instructional  Modes  Designated  for  CTS  Courses 

It  was  intended  that  the  amount,  type  and  degree  of  difficulty  of  both 
initial  instruction  and  remedial  instruction  would  be  influenced  by  the  student's 
past  performance.  As  a consequence  various  instructional  modes  were  designated 
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for  the  three  CTS  courses.  In  actuality  none  of  the  courses  completely  ful- 
filled the  requirements  for  the  designated  instructional  modes.  Described 
below  are  the  designated  instructional  modes  for  each  course  and  a discussion 
of  how  (if  at  all)  the  modes  were  implemented. 

(1)  31E  Course. 

In  the  31E  Course  there  were  three  designated  instructional  modes — 
high,  medium  and  low  performance.  The  level  of  performance  assigned  to  a student 
determines  the  amount  and  type  of  instruction  that  the  student  will  receive.  The 
higher  the  level  of  performance  the  less  detailed  is  the  instruction  received 
and  also  less  individual  attention  is  given  the  student  by  the  classroom  in- 
structor. Low  performers  are  generally  given  a greater  amount  of  instruction 
through  video  and  audio  means  than  are  the  high  performers.  All  students  enter 
the  course  labeled  as  a "medium"  performer.  The  CTS  automatically  monitors  the 
students'  performance  level  and  adjusts  it  as  appropriate.  Performance  level 
is  raised  (if  less  than  "high")  by  passing  three  consecutive  tests  on  the  first 
attempt.  These  tests  may  include  pretests,  posttests,  and/or  performance 
tests.  The  performance  level  is  lowered  as  a consequence  of  failing  two  con- 
secutive tests  (not  including  pretests). 

(2)  35L  Course 

In  the  35L  Course  the  same  three  instructional  levels  of  performance 
are  designated  as  for  the  31E  course.  However,  at  the  present  time  all  initial 
instruction  is  geared  to  the  medium  level  of  performance.  Remediation  instruc- 
tion is  the  only  situation  where  the  instructional  mode  concept  of  high,  medium 
and  low  performance  level  will  affect  instruction.  The  process  of  raising  or 
lowering  the  performance  level  is  the  same  for  the  35L  course  as  for  the  31E 
course. 
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(3)  31J  Course 


I 
I 

In  the  31J  Course  there  are  5 designated  instructional  modes.  However, 
only  instructional  mode  1 is  presently  available  for  implementation.  The  5 
modes  are  as  follows: 

Mode  1 consists  of  training  utilizing  a Performance  Guide  which  directs 
the  student  to  relevant  paragraphs  in  the  Technical  Manuals.  The  performance 
guide  also  directs  the  student  to  specific  video  tapes. 

Mode  2 will  consist  of  mode  1 instruction  supplemented  with  audio 

♦ 

instruction. 

Mode  3 will  include  the  instruction  of  modes  1 and  2.  In  addition, 
the  Performance  Guide  will  amplify  the  material  contained  in  the  Technical  Manuals 
(expanded  written  narrative  of  the  Technical  Jlanual). 

Mode  4 will  consist  of  Mode  3 instruction  supplemented  with  expanded 
audio  and  video  tape  instruction. 

Mode  5 is  intended  to  be  based  primarily  on  hands-on  training. 

According  to  the  CTS  design,  a mode  of  instruction  is  selected  for  a 
student  based  on  the  student's  average  Progression  Index'*’  (PI)  on  completed 
instruction.  If,  for  example,  a student  has  taken  Modes  1 and  2 and  Mode  2 has 
a lower  iMrogression  Index  (PI),  then  his  subsequent  instruction  within  the 
annex  will  be  in  Mode  2 until  and  unless  the  PI  in  that  mode  becomes  greater 
than  that  in  Mode  1. 


A discussion  of  the 


Progression  Index  is  in  the  next  Gection. 


I 
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Student  Progression  Index 

The  Student's  Progression  Index  (PI)  is  a ratio  of  POI  time  alloted  for 
a TAIS  or  Annex  and  the  actual  time  required  for  completion  of  the  TAIS  or 
Annex.  Pi's  are  automatically  computed  by  CTS  and  manually  computed  by  the  course 
personnel.  CTS  can  provide  two  different  measures  of  PI.  They  are:  portal-to- 
portal  PI  and  academic  PI.  Portal-to-portal  PI  is  used  time  divided  by  POI 
time.  Academic  PI  is  used  time  minus  absent  time  divided  by  POI  time.  In 
addition  to  maintaining  the  student's  PI  for  individual  TAIS's  and  Annexes,  a 
cumulative  PI  average  is  maintained  both  on-line  and  off-line. 

Generalized  Instructional  Process 
Prior  Training 

Before  entering  any  of  the  three  courses  the  student  will  have  completed 
a course  in  Common  Basic  Electronics  Training  (COBET) . This  course  is  self- 
paced  and  is  normally  given  at  Ft.  Jackson,  South  Carolina.  The  COBET  POI 
time  schedule  is  4 weeks  for  the  31E  Course,  6 weeks  for  the  31J  Course,  and  8 
weeks  for  the  35L  Course.  A lock-step  version  of  COBET  is  available  at 
USASC&FG  but  few  CTS  students  have  attended  this  version. 

Enrollment  of  Student  and  Creation  of  Student  Record 

Before  arrival  at  Ft.  Gordon,  the  student  will  have  been  preceded  by 
i TRADOC  Educational  Data  System  (TREDS)  records.  These  records  provide  name, 

social  security  number,  aptitude  area  scores,  and  various  background  informa- 
tion. TREDS  records  are  entered  into  CTS  by  the  Data  Systems  Division  (DSD) 
along  with  course  identification,  date  of  entry  into  course,  and  various 
other  needed  information.  All  counters  are  set  to  zero.  A class  roster  is 
printed  from  the  CTS  student  file  and  is  submitted  to  the  Course  Chief. 
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Student  Introduction  to  Course  and  CTS. 


A verbal  introduction  to  the  appropriate  course  is  given  by  the  Course 
Chief.  In  this  introduction  the  student  is  told  how  his  course  is  generally 
organized,  how  much  time  they  expect  him  to  take  in  completing  it,  when  the 
training  periods  are,  and  what  are  break  procedures  to  be  followed. 

Following  the  introductory  briefing  the  student  is  directed  to  the  class- 
room where  the  CTS  terminals  are  located.  The  classroom  instructor  gives  the 
student  a brief  verbal  introduction  to  CTS.  The  instructor,  explains  the  log-on 
procedure,  informs  the  student  of  his  ID  number,  asks  students  to  make  up  a 
password,  and  has  the  student  log  on  to  the  system  for  the  first  time. 

The  student's  first  interaction  with  CTS  is  a brief  lesson  dealing  with  the 
use  of  the  keyboard  and  an  explanation  of  the  progression  of  the  instruction. 

Introduction  to  Training  Materials 

The  master  off-line  student  control  document  is  the  Lesson  Study  Guide. 

This  guide  tells  the  student  the  materials  and  equipment  required  for  each 
learning  element.  The  guide  may  also  refer  the  student  to  appropriate  para- 
graphs in  TechnicalManuals  (CTS  sometimes  performs  this  function  also).  The 
guide  frequently  serves  as  a workbook  in  that  there  may  be  questions  or  exer- 
cises for  the  student  to  complete.  It  will  also  contain  checkpoints  at  which 
the  instructor  records  student  completion  of  a particular  study  activity.  All 
instruction  for  the  35L  Course  takes  place  in  the  CTS  classroom.  That  is,  each 
classroom  contains  its  own  audiovisual  and  video  tape  recording  (VTR)  equip- 
ment, as  well  as  the  CTS  terminals.  In  the  31E  and  31J  Courses,  the  audio- 
visual and  VTR  equipment  is  located  in  separate  Media  rooms.  At  appropriate 
times  in  the  instruction,  the  student  is  directed  to  leave  the  CTS  learning 
room  and  go  to  the  Media  room. 
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Administration  of  Basic  Electronics  Test 

The  student  logs  on  CTS  and  is  administered  a basic  electronics  test  to 
determine  his  state  of  knowledge  of  basic  electronics  and  areas  in  which  he 
requires  refresher  instruction.  If  the  student  displays  weaknesses  on  this  test 
he  is  provided  selective  remediation.  In  the  31E  and  31J  Courses  remediation 
will  usually  involve  study  of  selected  topics  at  the  Media  Center.  The  specific 
topics  for  study  are  defined  by  CTS  for  these  two  courses.  In  the  35L  Course 
the  instructor  prescribes  the  remediation  required  and  will  usually  administer 
the  remediation  instruction  himself.  In  all  three  courses  the  student  will  be 
re-administered  those  items  on  the  basic  electronics  test  that  he  has  failed 
at  the  initial  testing.  In  those  cases  where  the  student  again  fails  the  basic 
electronics  test  the  instructor  will  work  through  individual  items  to  determine 
and  correct  any  problems  in  understanding. 

Recording  Entry  Time  into  Annexes,  Tasks,  and  TAIS. 

In  all  CTS  courses  entry  times  into  Annexes,  Tasks  and  TAIS  are  recorded 
both  off-line  by  the  classroom  instructor  and  are  also  recorded  on-line  on 
the  CTS.  The  forms  used  for  recording  Entry  Time  (and  other  information)  and 
the  method  of  recording  varies  among  the  three  courses.  In  this  section 
only  the  35L  Course  is  described  as  reported  to  us  regarding  what  records 
are  maintained  and  how. 

Figure  7 is  a copy  of  a Student  Training  Record  in  the  35L  Course.  Each 
Task  within  the  course  will  have  its  own  Student  Training  Record  which  is 
maintained  by  the  classroom  instructor.  As  is  shown  on  the  sample  Student 
Training  Record,  all  TAIS's  within  the  Task  are  also  represented.  The 
instructor  records  the  actual  time  that  the  student  begins  instruction  in  the 
first  TAIS  of  the  course. 
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STUDENT  TRAINING  RECORD 


Figure  7.  Sample  Student  Training  Record. 


Entry  time  into  the  first  TAIS  of  the  course  is  recorded  as  beginning  as 
soon  as  the  student  begins  the  on-line  course  introduction.  The  first  TAIS 
includes  a course  introduction  and  the  Basic  Electronics  test,  with  multi- 
tracking strategy  such  as  the  35L  Course.  This  TAIS  is  included  in  the 
selected  entry  Task.  Since  the  Student’s  Progression  Index  (PI)  is  a ratio 
of  POI  time  allotted  for  a TAIS  and  the  actual  time  required  for  completing 
the  TAIS,  a method  is  needed  to  adjust  the  entry  Task.  The  method  used  for 
adjusting  the  first  PI  is  to  add  time  allotted  for  the  introductory  TAIS  to 
the  POI  time  allotted  for  completion  of  the  first  TAIS  in  the  entry  Task.''" 

Administration  of  Pretest,  When  Available 

As  was  stated  earlier,  there  are  no  pretests  for  TAIS  available  in  the 
35L  Course.  Pretests  are  available  for  some  TAIS’s  in  the  31E  and  31J  Courses. 
Students  have  the  option  of  taking  these  pretests  or  immediately  beginning 
the  instruction.  Actual  administration  of  the  pretest  has  already  been 
discussed.  However,  to  summarize,  if  the  student  passes  the  pretest  he  will 
advance  to  the  next  learning/performance  element.  If  he  fails  the  pretest,  he 
will  be  given  the  instruction  for  that  learning  element. 

Presentation  of  Instructional  Material 

Instruction  for  the  particular  learning  or  performance  element  in  the  TAIS 
may  be  presented  by  CTS  but  is  usually  provided  off-line.  CTS  and/or  the 
Lesson  Study  Guide  will  prescribe  the  specific  training  the  student  is  to  re- 
ceive. As  discussed  earlier,  initial  training  in  the  31J  and  35L  Courses  is  not 
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This  may  differ  in  the  other  CTS  Courses. 
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^ dependent  upon  the  mode  assignment  of  the  student  or  of  the  performance  level 

to  which  he  is  assigned.  However,  in  the  31E  Course,  the  student  begins 
instruction  in  the  first  TAIS  as  a medium  performer  and  will  be  raised  or 
lowered  based  on  his  performance  within  each  TAIS. 

I Upon  completion  of  instruction  for  all  learning/performance  elements  in 

the  TAIS  the  student  will  report  to  his  instructor.  The  instructor  determines 
if  the  student  has  spent  enough  time  studying.  If  so,  the  student  is  permitted 
to  sign  back  on  to  CTS  to  be  administered  the  posttest  for  that  learning  element. 

Administration,  Scoring  and  Recording  Posttest  for  Learning/Performance 
El ement 

I 

Although  strategies  used  in  the  three  CTS  courses  varied  somewhat,  for 
the  sake  of  brevity  we  will  describe  the  35L  strategy  reported  to  us.  After 
completing  the  training  for  a learning  element  within  the  TAIS,  the  student 
is  administered  an  on-line  posttest.  The  test  consists  of  15  items  which 
CTS  randomly  selects  from  a test  item  pool.  If  the  student  answers  80%  or 
more  of  the  questions  correctly,  he  will  have  passed  the  test.  Correct 
answers  are  then  given  to  items  incorrectly  answered.  CTS  automatically 
records  the  student's  grade  on  the  posttest  and  advances  the  student  to  the 
next  element.  The  instructor  also  records  on  the  Student  Training  Record  that 
the  student  has  completed  that  learning/performance  element. 

If  the  student  fails  the  posttest,  CTS  or  the  instructor  will  examine  the 
student  records  to  determine  if  the  previously  presented  test  (either  post- 
test or  performance  test)  was  also  failed.  If  so,  remediation  is  prescribed. 

If  the  student  fails  the  posttest  with  a score  between  50-69%,  CTS  will 
present  a message  telling  the  student  that  he/she  has  a chance  of  upgrading 
the  test  score  to  a passing  level.  Hints  will  be  given  on  questions  missed. 

These  hints  refer  the  student  to  specific  paragraphs  in  the  Technical  Manual. 
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The  student  then  reads  the  paragraphs  and  immediately  answers  the  question. 

There  is  little  chance  to  make  an  error.  If  the  student  does  not  obtain  a 
passing  level  of  70%  or  greater,  he/she  is  provided  remediation  and  retakes 
the  posttest. 

If  the  student  fails  the  posttest  with  fewer  than  50%  of  the  items 
answered  correctly,  CTS  will  inform  the  student  and  the  instructor  that 
remedial  instruction  is  needed.  The  instructor  prescribes  the  remedial 
instruction.  The  type  and  amount  of  remedial  instruction  will  be  determined 
by  the  student's  performance  level  (high,  medium,  or  low).  The  student  completes 
remedial  assignment  and  logs  back  onto  CTS.  The  instructor  verifies  on-line 
that  remediation  training  has  been  completed  and  the  student  is  administered 
a new  version  of  the  posttest.  Meanwhile,  the  instructor  will  post  the 
Student  Training  Record  indicating  that  remedial  instruction  was  administered. 

The  student  follows  the  above  procedures  until  the  posttest  has  been 
passed.  The  student  will  then  proceed  to  the  next  element. 

Administration,  Scoring  and  Recording  of  TAIS  Performance  Test 

After  the  student  has  completed  all  learning  elements  in  the  TAIS  he  or 
she  will  be  given  the  option  to  immediately  take  an  off-line  performance  test 
covering  the  material  in  the  entire  TAIS  or  to  receive  additional  practice  be- 
fore taking  the  test.  If  the  student  opts  to  take  the  test  the  CTS  keyboard 
locks  and  the  message  "Select  and  administer  a lesson  test  version  from 
Supervisor's  Guide"  is  shown  on  the  screen  to  the  instructor.  The  instructor 
then  selects  one  of  several  test  versions.  Each  version  will  present  a particular 
set  of  malfunctions  in  theequipment  under  study.  A typical  TAIS  Performance 
Test  could  consist  of  four  items.  The  student  must  get  at  least  two  of  the 
four  items  correct  to  pass  the  test.  The  student  is  permitted  a maximum  of  ten 


minutes  for  each  problem.  .The  student  is  graded  on  each  question  before  going 
on  to  the  next  question.  If  the  student  is  incorrect  on  a problem,  the  in- 
structor will  consider  this  a "NO  GO"  but  will  critique  the  problem  with  the 
student  before  going  on  to  the  next  test  problem.  If  the  student  fails  the  test 
(has  a NO  GO  on  3 or  A items)  the  instructor  selects  and  gives  appropriate  re- 
medial instruction.  Following  this  instruction  the  test  is  readministered.  This 
procedure  is  followed  until  the  student  has  passed  the  test. 

After  the  student  has  passed  the  test  the  instructor  will  post  the  com- 
pletion time  for  the  TAIS  on  the  Student  Training  Record.  The  instructor  will 
also  record  on-line  on  CTS  the  version  used  to  test  the  student,  and  the  number 
of  NO  GO's  the  student  received. 

The  student  advances  to  the  next  TAIS  and  completes  the  study  and  test 
requirements  within  that  TAIS.  When  all  TAIS's  have  been  completed  in  the  Task, 
the  student  will  be  administered  an  EPIS  test. 

Administration,  Scoring  and  Recording  of  EPIS  Test 

After  completing  all  TAIS's  within  a Task  the  student  is  administered  an 
EPIS  off-line  performance  test  covering  all  skills  taught  in  the  Task.  The 
test  is  administered  by  the  instructor  in  the  35L  classroom;  in  the  31E  and 
31J  Courses  it  is  administered  in  a special  testing  center.  Prior  to  the  test 
the  instructor  has  the  student  leave  the  CTS  area.  CTS  then  provides  the  in- 
structor with  directions  for  test  administration  and  with  a randomly  selected 
sample  of  malfunctions  for  testing.  The  instructor  may  reject  this  sample  and 
obtain  another  sample  of  malfunctions  from  CTS.  The  specific  sample,  or  samples, 
of  malfunctions  are  permanently  recorded  on  CTS.  The  instructor  then  recalls 
the  student  to  the  CTS  where  the  student  is  given  instructions  for  taking  the 
test.  The  student  is  then  automatically  signed  off  CTS  and  proceeds  to  take  the 
test  in  the  same  way  that  he  took  the  TAIS  performance  test;  the  instructor 
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responsible  for  testing  judges  and  records  the  student's  performance.  After 
completing  the  entire  test  the  student  signs  back  onto  CTS  and  is  told  to  call 
his  instructor  to  the  CTS  station.  The  instructor  records  the  following  in- 
formation: instructor  password,  test  ID  number,  time  allotted  for  test, 

whether  the  test  was  passed  or  failed  (GO  NOGO) , and  how  each  individual  item 
was  scored.  This  information  is  permanently  stored  in  CTS  records. 

If  the  student  has  failed  the  EIPS  test,  the  instructor  determines  the 
type  and  amount  of  remediation  the  student  will  receive  before  readministration 
of  the  test. 

At  this  time,  the  instructor  updates  the  Student  Training  Record  and 
telephones  his  supervisor  to  determine  which  Task  the  student  is  to  be  next 
assigned.  The  Student  Training  Record  is  forwarded  to  the  supervisor  who 
uses  the  information  contained  on  it  to  post  a Summary  Training  Record  (see 
Figure  8) . 

Course  Completion 

The  student  completes  the  course  after  successfully  completing  training 
in  all  Annexes  of  the  course.  Four  and  two  weeks  prior  to  the  course  completion 
date,  DA  is  notified  as  to  when  the  student  will  be  available  for  reassignment. 
On  the  same  day  that  the  student  completes  training,  he  or  she  will  receive  a 
certificate  of  completion  and  will  be  returned  to  his  or  her  unit  for  out- 
processing. 
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8.  How  useful  are  the  CTS-generated  reports  for  instructors  and 
training  managers? 


The  Operational  Reports  Survey  form  was  administered  to  33  USASC&FG 
instructional  personnel.  Fifteen  responses  were  received. 

Thirteen  of  the  CTS-generated  reports  were  presented  for  comment  on  this 
form.  The  results  of  this  Survey  form  are  found  in  Table  10.  Of  the  13 
reports,  6 were  known  to  only  20%  or  less  of  the  respondents.  These  were 
the 

• Student  Evaluation  Roster 

• Course/Task  Report  (Monthly) 

• TAIS  Report  (Monthly) 

• Test  Analysis  (Monthly) 

• Question  Analysis — Districtor  Count  (Monthly) 

• Question  Analysis — Constructed  Response  Printout  (Monthly) 
However,  those  few  instructional  personnel  who  were  familiar  with  these 

forms  regarded  them  very  positively.  Thus,  it  appears  that  the  availability 
and  utility  of  these  reports  needs  to  be  communicated  to  the  USASC&FG  staff  in 
order  that  the  reports  be  accepted  and  used. 

Of  the  other  7 reports,  the  Student  Class  Roster  was  received  very 
favorably  with  only  13%  of  the  respondents  of  the  opinion  that  there  were 
numerous  errors  in  the  form,  or  that  the  format  required  revision.  On  the 
other  hand,  all  of  those  respondents  familiar  with  this  form  considered  the 
information  presented  in  this  form  as  essential.  A similar  reaction  was  given 
by  those  respondents  familiar  with  the  TAIS  Report  (Monthly) . Sixty  percent 
of  the  respondents  were  familiar  with  the  Meekly  Student  Activity  Report.  Of 
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these,  less  than  10%  had  negative  reactions  to  it.  The  same  response  break- 
down was  obtained  regarding  the  Graduation  Prediction  and  Graduation  Reports 
\ The  monthly  Course  Absentee  and  Company  Absentee  Reports  were  familiar 

^ I to  40%  of  the  respondents.  Of  these,  less  than  half  were  not  satisfied  with 


the  reportsas  currently  formatted  and  the  information  they  contained. 

All  of  the  findings  discussed  above  are  based  on  a respondent  population 
of  15.  The  responses  to  our  structured  interview  amplify  these  findings. 
Although  most  of  the  CTS-generated  forms  were,  by  and  large,  of  use  to 
instructional  personnel,  most  stated  that  they  were  unaware  of  their  avail- 
ability. This  was  probably  based  on  their  mistrust  of  the  data  and  their 


reliance  on  the  manual  records/reports  to  which  they  had  been  accustomed. 


i 
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Table  10.  Responses  to  Operational  Reports  Survey  (5  of  Respondents) 
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STUDENT  RECORD  PRINTOUT 
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9.  Has  the  System  been  effective  in  monitoring  students  within  the 
respective  Tasks  and  Annexes? 


This  question  was  answered  by  means  of  eleven  items  from  the  Resource 
Allocation  Survey.  The  items,  presented  primarily  to  instructors  from  the 
three  CTS  courses,  are  organized  into  three  groups  shown  in  Table  11. 

The  first  group,  consisting  of  five  items,  deals  with  normal,  uninterrupted 
student  flow.  The  second  group,  consisting  of  four  items,  deals  with  student 
flow  when  insufficient  practice  and  test  equipment  is  available  to  handle 
students  as  they  complete  one  task  and  are  ready  to  begin  the  next.  The  third 
group  (two  items)  concerns  instructor  workload.  The  responses  to  the  three 
types  of  items  are  summarized  in  Table  11. 

The  data  indicate  that  under  normal  student  flow,  the  Instructors  reported 
that  the  system  can  satisfactorily  route  the  student  from  task  to  task  and 
through  the  entire  course,  determining  and  revealing  where  the  student  is,  with 
few  problems  or  errors.  It  may  be  seen  that  the  majority  of  responses  indicate 
agreement  with  statements  concerning  CTS  capacity  to  perform  these  functions. 

However,  responses  to  items  in  the  second  group  suggest  that  instructors 
perceive  the  system  to  be  less  capable  of  routing  students  when  demand  for 
equipment  and  practice  stations  increases  beyond  normal.  For  those  items 
dealing  with  system  tracking  of  equipment  availability,  routing  of  students  to 
alternate  tasks,  and  reporting  a shortage  of  equipment,  responses  shift  from 
largely  favorable  to  largely  neutral  or  negative.  Furthermore,  the  instructors 
indicate  in  the  last  question  of  this  group  that  the  responsibility  for  routing 
students  falls  on  them. 
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The  two  items  comprising  the  third  group  deal  with  the  instructor's 
role  of  monitoring  student  flow  before  CTS  and  after  its  implementation.  The 
first  item  indicates  that  the  instructors  were  about  equally  divided  in  their 
belief  that  accounting  for  student  time  prior  to  CTS  was  an  unnecessary  burden. 
The  second  item  indicates  that  they  remain  equally  divided  on  whether  CTS 
usefulness  in  monitoring  student  flow  offsets  associated  demands  on  instructors. 


Table  11.  Resource  Allocation  Survey  Computerized  Training  System  (CTS) 


Group  I : Student  Flow,  Unrestricted  Resources 

The  computer  system  has  been  able  to  accurately 
account  for  the  location  of  each  student  within  the 
course . 

When  the  student  completes  a task,  the  system  routes 
him  to  the  next  task  without  delay. 

Students  are  routed  through  tiie  course  according 
to  the  predetermined  (normal)  flow. 

Student  routing  has  been  accomplished  with  a 
minimum  of  errors. 

The  instructor  is  alerted  via  the  hard  copy 
terminal  when  the  student  lias  completed  all  the 
required  tasks. 


Group  II:  Student  Flow,  Restricted  Resources 

The  computer  system  has  maintained  an  accurate 
accounting  of  student  position  vacancies. 

When  student  positions  in  the  next  sequential 
task  are  filled,  the  student  is  routed  to  an 
alternate  task,  fqr  which  he  has  the  necessary 
prerequisites. 

When  the  student  completes  a task  and  all  student 
positions  in  the  succeeding  tasks  are  filled,  the 
instructor  is  alerted  via  the  hard  copy  terminal . 

Routing  students  to  the  correct  student  positions 
has  required  close  monitoring  by  the  instructor/ 
supervisor . 


Group  III:  Instructor  Support 

Prior  to  CTS,  did  accounting  for  student  time  in  the 
course  place  an  unnecessary  burden  on  the  classroom 
instructor? 

Does  the  usefulness  derived  from  using  the  computer 
for  accounting  for  students'  time  in  the  CTS  tasks 
outweigh  the  workload  Inherent  in  its  collection? 
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Strongly  Disagree 


10.  Were  sufficient  resources  available  to  handle  the  student  load? 


The  Instructional  Process  Survey,  designed  to  answer  this  question, 

could  not  be  used  because  only  two  respondents  returned  completed  instruments. 

( 

However,  during  extended  site  visits  to  a]l  three  courses,  it  was  evident  to 
HumRRO  personnel  that  little  or  no  queueing  occurred  at  the  computer  terminals 
during  class  sessions.  Students  were  able  to  interact  with  the  CTS  whenever 
required  by  the  student  manual  or  by  the  instructor.  Further,  interviews 
with  instructional  programmer  personnel  suggested  that  available  CTS  resources 
had  no  adverse  affect  on  the  courses'  ability  to  handle  student  throughput. 
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11.  I 8 there  an  adequate  back-up  capability  to  provide  instruction 
during  computer  down- time? 


Information  concerning  back-up  capability  could  not  be  obtained  from  the 
Instructional  Process  Survey  as  originally  planned  because  of  the  almost 
total  lack  of  response  to  that  instrument.  Classroom  observation  and  structured 
interviews  of  CTS  personnel  by  HumRRO  scientists  indicate,  however,  that 
available  back-up  materials  were  sufficient  to  ensure  the  continuation  of 
instruction  during  computer  down-time.  Hardcopy  versions  of  on-line  tests 
enabled  instructors  to  administer,  score,  and  record  pre-  and  posttests; 
the  instructors  could  give  assignments  based  on  their  familiarity  with  the 
course  content  and  the  instructional  strategy.  It  is  possible  that  had  on-line 
instruction  comprised  a higher  percentage  of  each  of  the  courses,  off-line 
back-up  may  have  been  overtaxed.  Instructors  reported,  however,  that  since 
on-line  instruction  comprised  only  11-12%  of  the  courses,  they  could  handle 
students  at  those  times  when  the  computer  was  down. 


Supervisory/Managernent  Comments  Related  to  Course  Administration 


Many  supervisory  respondents  used  HumRRO's  structured  interview  as  an  op- 
portunity to  discuss  problems  they  perceived  or  experienced  during  their  in- 
volvement with  CTS.  A concern  expressed  consistently  across  the  large  majroity 
of  supervisors  associated  with  all  three  courses  was  the  diversion  of  instructor 
personnel  to  CTS-related  duties  with  no  provision  for  additional  help  in  other 
ongoing  training  responsibilities.  A typical  comment:  "Instructional  programming 
came  out  of  our  hide,  which  meant  that  there  was  an  increased  burden  on  other 
instructors."  Another  respondent  remarked  on  a "need  for  authorized  staffing 
for  CTS;  (we  are)  using  instructors,  which  means  that  less  manpower  is  available 
for  other  tasks."  These  problems  occurred  even  though  the  CTS  field  office 
supported  and  augmented  the  USASC&FG  course  development  effort  initially  with 
9 people. 

In  a related  area,  several  supervisors  remarked  on  the  problems  associated 
with  personnel  turnover;  their  feeling  was  that  trained,  experienced  personnel 
did  not  stay  on  the  job  long  enough  to  provide  continuity  to  the  instructional 
development  process.  One  Division  Chief  expressed  the  opinion  that  "experienced, 
key  personnel  should  have  stayed  with  the  course,  but  military  personnel  stay 
for  a maximum  of  18  months  and  civilians  look  for  promotion." 

Another  frequently  voiced  concern  was  over  the  apparent  "inaccuracy"  of 
the  CTS-maintained  records  on  training  time,  absence  time  and  progression  indexes. 
Supervisors  in  two  of  the  CTS  courses  cited  discrepancies  between  the  length 
of  an  instructional  period  as  employed  in  the  course  and  that  employed  by  the 
computer  system  to  compute  training  time.  Specifications  for  length  of  the 
instructional  period  were  not  kept  current.  Thus,  the  progression  Indexes 
computed  manually  by  instructors  differed  from  those  generated  by  the  CTS  system. 
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Others  complained  of  the  difficulty  encountered  In  maintaining  accurate  absence 
time  in  the  CTS  system.  One  respondent  remarked  that  it  was  difficult  to 
enter  absence  data  into  the  system.  If  a student  had  progressed  to  a new  Task 
or  TAIS  before  a valid  absence  time  could  be  entered,  the  system  would  not 
accept  the  data.  To  the  extent  that  absence  entries  were  incomplete,  the 
system-recorded  training  times  and  computed  progression  indexes  would  be 
inaccurate. 

Other  comments  by  supervisors  included:  "They  shouldn't  experiment 
using  an  ongoing  training  course";  "there  was  turmoil  when  the  CTS  conversion  was 
added  to  the  requirement  for  self-pacing";  "management  wouldn'r  become  involved 
with  CTS  problems";  "cheap  hardware  and  not  enough  storage";  and  "there  were 
problems  with  the  software." 

HumRRO’s  interviewers  were  struck  by  the  absence  of  positive  comments 
on  the  part  of  management  personnel  concerning  CTS.  Unlike  the  instructional 
programmers  who  appeared  involved  and  to  some  extent  dedicated  to  making  CTS 
work,  the  supervisors  gave  the  strong  Impression  that  CTS  was  viewed  as  an 
impediment  to  the  school's  principal  goal  of  turning  out  course  graduates  in 
the  time  expected  and  numbers  required. 
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4.  Training  Effectiveness 

In  this  section  we  will  describe  and  discuss  those  evaluation  questions 

! 

which  deal  with  the  training  effectiveness  of  CTS  (Questions  15-18). 
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15.  What  is  the  average  student  progression  index  by  Task  and  Annex 
for  each  CTS  course?  What  is  the  standard  deviation?  Hew  many 
graduates  were  there  in  eaah  course? 


31E  Course 

CTS  personnel  estimated  that  approximately  11.7%  of  all  the  CTS  course 
material  was  on-line.  Of  this  material,  most  was  made  up  of  pre-  and  post- 
tests. For  the  31E  course,  60  hours  or  approximately  8.7%  of  the  691  POT  hours 
were  on-line.  The  actual  training  hours  on  the  average  in  the  CTS  version  of  the 
31E  course  (764)  was  10.6%  more  than  that  allocated  by  the  POI.  The  Self-Paced 
31E  Course  training  time  was  84%  of  that  allocated  by  the  POI  (653  hours). 

It  was  decided  that  those  tasks  with  different  POI  hours  in  the  CTS  and 
Self-Paced  versions  of  the  courses  would  be  removed  from  any  comparative 
analyses  (Tasks  1,  7 and  10  in  the  31E  Course).  Although  one  could  compare 
Pi's  indepedently  of  POI  hours,  it  was  felt  that  POI  hour  changes  might  have 
reflected  substantive  content  alterations.  The  effects  that  these  modifications 
might  have  had  on  training  time  are  not  known  and,  thus,  we  felt  should  Be 
deleted  from  any  comparative  analyses.  We  have  presented  all  the  31E  data  in 
Table  12,  however,  to  provide  the  reader  with  a comprehensive  picture  of  our 
study's  findings. 

Comparisons  were  made  between  the  CTS  and  Self-Paced  actual  training  times 
expended  by  Task  (for  those  Tasks  with  identical  POI  hours).  The  results  show 
that  the  CTS  version  was  significantly  more  efficient  only  in  Task  9.  Self- 
paced  training  times  were  significantly  lower  than  CTS  in  Tasks  2,  3,  4,  8,  11, 
13,  and  14.  No  significant  differences  were  found  for  the  remaining  Tasks  (5, 

6,  and  12).  The  overall  comparison  for  these  Tasks  (without  1,  7 and  10)  showed 


I 

I 
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Table  12..  Training  Time  Data--CTS  and  Self-Paced  31E  Course 


Academic 

POI 

Actual  Hours 

to  Complete 

Progression 

Index 

Tasks 

Hours 

X 

S.D. 

x S 

.D. 

JL 

CTS 

101 

116.6 

43.2 

1.16 

.43 

46 

Self-Paced 

87 

85.6 

18.6 

.98 

.21 

59 

CTS 

54 

33.1. 

20.4 

•61* 

.38 

46 

Self-Paced 

54 

25.9 

8.6 

.48* 

.16 

59 

CTS 

36 

39.7. 

12.6 

1.10* 

.35 

46 

Self-Paced 

36 

24.8 

8.2 

.69 

.23 

59 

CTS 

59 

1.34.8. 

42.1 

2.28. 

.71 

46 

Self-Paced 

59 

113.2 

24.5 

1.92 

.42 

59 

CTS 

69 

34.7 

12.5 

.50 

.18 

46 

J 

Self-Paced 

69 

34.7 

11.7 

.50 

.17 

59 

CTS 

68 

66.9 

88.0 

.98 

.17 

46 

0 

Self-Paced 

68 

70.7 

43.2 

1.04 

.64 

59 

7 

CTS 

70 

120.8 

54.7 

1.73 

.78 

46 

Self-Paced 

56 

64.7 

50.7 

1.16 

.91 

59 

8 

CTS 

50 

24.4. 

8.9 

•49* 

.18 

46 

Self-Paced 

50 

18.9 

12.9 

.38 

.26 

59 

Q 

CTS 

33 

10.0. 

5.8 

•30* 

.18 

46 

Self-Paced 

33 

30.0 

24.6 

.91 

.74 

59 

10 

CTS 

35 

92.9 

58.6 

2.52  1 

.40 

46 

Self-Paced 

25 

27.0 

20.9 

1.08 

.84 

59 

11 

CTS 

Self-Paced 

23 

23 

9-9* 

8.2 

4.4 

3.5 

•«* 

.36 

.19 

.15 

46 

59 

12 

CTS 

24 

13.2 

6.7 

.55 

.28 

46 

Self-Paced 

24 

13.0 

7.6 

.54 

.32 

59 

13 

CTS 

24 

47.3. 

18.5 

1.97 

.77 

46 

Self-Paced 

24 

19.9 

18.9 

.83* 

.79 

59 

14 

CTS 

45 

19.9. 

6.7 

.44 

.15 

46 

Self-Paced 

45 

13.0 

5.9 

.29 

.13 

59 

Course:  31E 

CTS 

691 

764.2 

— 

1.11 

— 

46 

Self-Paced 

653 

549.4 

— — 

.84 

—— 

59 

Significantly  different  at  « ■ .05,  2-tailed 
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that  the  CTS  version  took  a significantly  longer  time  for  the  students  to 
complete  than  the  Self-Pace.d  version.  The  ability  level  of  the  student 
population,  however,  was  not  significantly  different  (as  measured  by  GT 
scores — CTS  GT  x = 114;  Self-Paced  x = 112) . Thus,  the  course  context/format 
appears  to  be  the  sole  contributing  factor  for  these  findings.  The  differences 
between  CTS  and  Self-Paced  training  time  appear  to  be  mostly  due  to  the  pre- 
and  posttests  that  were  part  of  the  CTS  courses.  A serious  question  is  raised 
regarding  the  efficacy  of  a pre-/posttest  instructional  design  if  the  course 
is  performance-oriented  (with  criterion  performance  tests),  and  the  students' 
entering  state  of  knowledge  is  quite  low.  In  the  latter  case,  all  students 
receive  the  same  instruction,  no  matter  what  the  pretest  outcome.  This 
situation  surely  contributed  to  the  relatively  slow  progress  of  the  CTS  group. 

31J  Course 

In  Table  13  we  show  the  time  data  obtained  from  the  31J  Course.  However, 
as  indicated  elsewhere,  the  students  going  through  the  CTS  course  were  in 
the  process  of  validating  training  materials  (none  of  the  students  completed  a 
fully  validated,  operational  31J  CTS  course).  In  fact,  to  this  report's  date, 
only  8 Tasks  of  the  31J  course  were  put  in  the  CTS  version  with  no  intention 
on  the  part  of  course  personnel  to  complete  this  conversion.  Therefore,  it 
was  felt  to  be  inappropriate  for  any  comparative  data  analyses  to  be  performed 
on  this  course. 

35L  Course 

In  Table  14,  we  show  the  time  data  obtained  from  the  35L  Course.  As 
can  be  seen,  not  one  Task  had  the  same  allocation  of  POI  hours  in  the  CTS  and 
Self-Paced  versions.  Therefore,  no  direct  comparisons  of  Task  training  times 
in  this  course  were  made.  Overall,  the  Pi's  for  CTS  and  Self-Paced  versions 
do  not  appear  to  be  different. 
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Table  13.  Training  Time  Data--CTS  and  Self-Paced  31J  Course 


Academic 

POI 

Actual  Hours 

to  Complete 

Progression  Index  ! 

Tasks 

Hours 

X 

S.D. 

X 

S.D. 

N 

CTS 

94 

119.6 

34.4 

1.27 

.37 

37 

1 

Self-Paced 

94 

86.0 

13.4 

.92 

.14 

17 

'i 

CTS 

50 

89.0 

32.8 

1.78 

.66 

36 

Self-Paced 

56 

66.7 

21.6 

1.20 

.39 

18 

CTS 

38 

32.9 

11.2 

.82 

.23 

37 

39 

68 

Self-Paced 

39 

65.7 

18.7 

1.02 

.24 

17 

68 

CTS 

60 

66.3 

17.5 

1.11 

.29 

33 

Self-Paced 

60 

108.3 

44.2 

1.33 

.47 

21 

85 

CTS 

85 

64.6 

0 

.76 

0 

1 

7 

Self-Paced 

137 

149.5 

31.7 

1.09 

.23 

17 

11 

CTS 

89 

56.1 

0 

.63 

0 

1 

Self-Paced 

89 

63.0 

15.8 

.70 

.18 

35 

90 

13 

CTS 

83 

N/A 

N/A 

N/A 

N/A 

N/A 

Self  Paced 

80 

74.9 

74.9 

.93 

.28 

36 

83 

15 

CTS 

44 

34.4 

4.8 

.78 

.09 

36 

Self-Paced 

44 

31.5 

8.5 

.72 

.23 

18 

Course:  31.1 

CTS 

543 

462.9 

— 

1.011 

— 

Self-Paced 

599-657 

645.6 

— 

1.08-.98 

— 

^Based  on  POI  hours-460 

(543  minus  83; 

adjusted  for 

no  CTS  data 

on  Task  13.) 
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Table  14..  Training  Time  Data--CTS  and  Self-Paced  35L  Course 


POI 

Actual  Hours 

to  Complete 

Academic 

Progression  Index 

Tasks 

Hours 

X 

S.D. 

X 

S.D. 

N 

. CTS 

122 

108.1 

27.9 

.90 

.23 

49 

Self-Paced 

74 

66 . 6 

22.9 

.90 

.31 

36 

2 CTS 

112 

1 114 

96.6 

24.6 

.88 

.24 

34 

Self-Paced 

77 

89.7 

25.2 

1.17 

.33 

36 

„ CTS 

80 

93.2 

28.6 

1.17 

.36 

54 

Self-Paced 

98 

76.0 

18.1 

.78 

.19 

36 

4 CTS 

73 

68.9 

15.4 

.94 

.21 

54 

Self-Paced 

76 

75.9 

20.1 

1.00 

.26 

36 

P CTS 

80 

84.9 

26.3 

1.06 

.33 

54 

Self-Paced 

72 

71.9 

23.2 

1.00 

.32 

36 

Course:  35L 

CTS 

467-469 1 

451.6 

— 

.97-. 96 

— 

Self-Paced 

397 

380.1 

.96 

1 Course  POI  = 507  hours;  performance  data  on  38  POI  hours  (Test  Equipment) 


not  available. 
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16.  What  was  the  number  and  percentage  of  students  who  failed  to 
graduate  from  eaeh  CTS  course ? 

< 

The  data  we  received*'  indicate  that  there  were  305  graduates  of  the  31E 

CTS  course  in  FY  1977.  During  that  year,  there  were  22  academic  losses.  In 

the  period  October  1977-March  1978,  an  additional  97  personnel  graduated  this 

course  and  22  attrited  for  academic  reasons.  In  the  35L  course,  70  CTS 

students  graduated  in  the  period  July  1977-May  1978.  Eleven  (11)  students 
failed  to  graduate  in  that  period.  Only  8 students  were  listed  as  having 
graduated  the  CTS  31J  course,  although  only  a portion  of  that  course  has  been 
put  on  CTS.  The  31J  training  time  data  that  we  used  came  from  students  during 
validation  of  the  training  materials.  This  reinforces  the  contention  that  only 
in  the  31E  and  35L  courses  was  the  training  system  stable  enough  for  some 
meaningful  data  analyses  to  be  performed;  and  only  in  the  31E  course  were  POI 
hours  stable  enough.  Therefore,  in  the  31J  course,  it  appears  that  all  student 
performance  data  are  highly  tenuous  and  should  not  be  considered  as  providing 
a basis  from  which  to  draw  any  conclusions. 


Personal  communication. 


17.  What  are  student  attitudes  towards  CTS  courses? 


Student  attitude  data  were  gathered  both  before  and  after  training  in 
the  31E  course.  The  attitude  data  revealed  no  consistent  differences  between 
initial  and  final  administration.  Therefore,  we  will  concentrate  our  discussion 
only  on  the  final  set  of  data.  Anonymous  responses  were  obtained  from  34 
students  on  the  post-course  attitude  survey.  The  student  attitude  questionnaire 
containing  summary  data  is  included  as  Attachment  #9.  Sixty-two  (62)  of  the 
79  items  required  the  respondent  to  rate  a statement  on  a five-point  scale 
whith  ranged  from  "strongly  agree"  to  "strongly  disagree."  The  remaining  items 
solicited  background  information,  student  perspectives  about  on-line 
instructional  time,  and  open-ended  questions  related  to  course  specifics. 

Nine  students  (out  of  the  34)  had  some  prior  CBI  experience . The  question- 
naire is  divided  into  four  sections.  The  four  sections  are: 

Section  I - items  related  to  instructional  content  and  media,  and  the 
learning  environment,  including  both  physical  and  support 
characteristics . 

Section  II  - items  related  to  the  computer  management  aspect  of  the 

training  experience  including  both  environmental  and  peda- 
gogical items. 

Section  III  - items  related  to  scheduling  of  on-line  time. 

Section  IV  - student  likes  and  dislikes  about  the  system. 


SECTION  I RESULTS 


Instructional  Materials.  For  those  items  which  were  answered  definitely 
(not  "neutral"),  the  students  felt  that  the  instruction  was  too  hard  (possibly 
because  the  reading  level  is  too  high  (item  8)  and  the  step  size  was  too  large. 
Students  reported  using  the  pre-and  post-test  as  a self -management  tool  to 
obtain  more  Instruction.  However,  they  did  find  the  material  interesting  and 
the  course  objectives  clearly  stated.  About  one-third  of  the  students  would 
prefer  more  instruction  delivered  via  CTS,  while  the  majority  would  not. 

Support . The  students  found  the  instructors  to  be  knowledgeable,  but 
not  always  available  when  required.  The  responses  to  item  18  indicate  that 
the  students  felt  isolated  and/or  ignored  while  working  with  CTS. 

Learning  Environment.  Generally  the  students  liked  to  use  the  computer 
terminals  as  an  instructional  device.  They  were  dissatisfied,  however,  with 
the  plethora  of  media  which  they  had  to  use. 

SECTION  II  RESULTS 

Instruction . The  directions  to  the  student  were  perceived  as  easy  to 
follow.  The  instructors  referred  the  students  to  the  computer  for  directions, 
rather  than  providing  them  themselves. 

Hardware.  The  students  felt  that  the  combination  of  computer  down-time 
and  lack  of  availability  of  computer  terminals  impeded  their  utilization  of 
the  systems. 

SECTION  III  RESULTS 

Schedul ing . Most  students  spent  an  average  of  20  to  30  minutes  at  the 

computer  terminal  in  a single  session  that  coincides  with  th^  time  in  wh  ch 

t 

they  get  uncomfortable  before  taking  a break.  However,  when  disked  How  lo  % 


others  should  work  at  a terminal  without  a break,  they  recommended  30  to  50 
minutes,  and  if  breaks  were  permitted — one  hour.  The  maximum  time  recommended 
for  a student  to  spend  at  a terminal  without  a break,  is  45  minutes.  Again, 
this  is  longer  than  the  respondents  themselves  spent  at  the  terminal.  Most 
students  were  willing  to  spend  up  to  three  hours  per  day  Interacting  with  the 
computer . 

SECTION  IV  RESULTS 

What  the  students  liked  best — the  rank  order  of  the  most  preferred  aspects 
of  computerized  instruction  is  shown  below: 

- I can  go  at  my  own  speed. 

- It  presents  materials  in  a clear  and  interesting  way. 

- I am  always  being  asked  questions. 

- I like  the  freedom  offered  by  CTS. 

- I am  not  bothered  by  an  instructor  except  when  I need  him. 

What  the  students  liked  least — the  rank  order  of  the  least  desirable  as- 
pects of  computerized  instruction  is  shown  below: 

- I cannot  ask  questions. 

- It  leaves  out  too  much  information  that  an  instructor  would  provide. 

- I have  to  learn  to  operate  too  much  instructional  equipment. 

- It  is  too  impersonal. 

- It  is  too  much  work. 
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18.  What  is  the  relationship  between  student  scores  on  CTS  administered 
task  testa  and  Progression  Index  values?  What  percentage  of  students 
passed  CTS  tests  on  the  first  try? 

Tables  15,  16,  and  17  show,  for  each  course,  the  percentage  of  students 
passing  Task  (EPIS)  tests  in  one,  two,  and  three  or  more  attempts.  Pass  rates 
are  presented  in  all  cases  for  CTS  administered  Tasks;  where  available,  compar- 
able data  are  presented  for  Self-Paced  Tasks. 

Table  15  presents  findings  for  the  CTS  students  in  the  31E  course;  no  in- 
formation was  available  on  Self-Paced  students.  It  can  be  seen  from  the  table 
that  at  least  80%  of  the  CTS  students  passed  the  majority  of  the  Task  tests  on 
the  first  try.  In  the  last  five  Tasks  of  the  31E  course,  the  percentage  of 
first-try  success  ranges  from  59%  (Task  10)  to  74%  (Task  14)  with  most  of  these 
values  clustering  around  seventy  percent. 

Table  16  shows  that  lor  the  31J  course  a generally  higher  percentage  of 
CTS  students  passed  the  Task  tests  on  the  first  try.  In  the  case  of  Task  15, 
the  first  attempt  success  rate  was  100%  for  both  CTS  and  Self-Paced  students. 
However,  in  the  remaining  Tasks,  the  first  attempt  passing  percentage  of  CTS 
students  ranged  from  81%  (Task  5)  to  97%  (Task  3),  while  among  Self-Paced 
students  the  first  attempt  passing  percentages,  with  the  exception  of  Task  1, 
were  all  substantially  less  than  80%. 

For  the  35L  course,  Table  17  shows  that  the  first  attempt  pass  rate  was 
generally  high  for  both  CTS  and  Self-Paced  students  on  all  comparable  Tasks. 

In  only  one  Task  (Task  4)  was  there  a notable  difference;  in  this  case  the 
value  is  100%  for  the  CTS  students  and  72%  for  those  in  the  Self-Paced  version. 

The  question  concerning  the  relationship  between  performance  on  Task 
tests  and  Task  Progression  Indices  (Pi's)  for  CTS  students  was  raised  because 
the  latter  measure  has  been  prominent  as  a benchmark  of  success  to  both 
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Table  15.  Percentage  of  CTS  and  Self-Paced  Students  Passing  31 E EIPS  Tests 
on  the  First,  Second,  and  Third  (or  higher)  Attempt 


CTS 



Sel f-Paced^ 

Task 

Attempt 

Percent 

N 

1 

1 

87 

(40) 

2 

11 

( 5) 

3+ 

2 

( 1) 

2 

1 

89 

(41) 

2 

9 

( 4) 

3+ 

2 

( 1) 

3 

1 

87 

(40) 

2 

13 

( 6) 

3+ 

0 

( 0) 

4 

1 

87 

(40) 

2 

11 

( 5) 

3+ 

2 

( 1) 

5 

1 

76 

(35) 

2 

22 

(10) 

3+ 

2 

( 1) 

6 

1 

87 

(40) 

7 

1 

86 

(39) 

2 

6 

( 3) 

3+ 

9 

( 4) 

8 

1 

83 

(38) 

2 

13 

( 6) 

9 

1 

80 

(37) 

2 

13 

( 6) 

3+ 

2 

( 1) 

10 

1 

59 

(27) 

2 

15 

( 7) 

3+ 

26 

(12) 

11 

1 

72 

(33) 

2 

22 

(10) 

3+ 

2 

( 1) 

1 


No  Self-Paced  31E  data  regarding  EIPS  tests  were  available. 


101 


continued 


* j 


Table  15  (continued).  Percentage  of  CTS  and  Self-Paced  Students  Passing 

31 E EPIS  Tests  on  the  First,  Second  and  Third  (or  higher)  Attempt. 


CTS 

Sel f-Paced^ 

Task 

Attempt 

Percent 

N 

12 

1 

67 

(31) 

2 

13 

( 6) 

3+ 

15 

( 7) 

13 

1 

67 

(31) 

2 

* 13 

( 6) 

3+ 

15 

( 7) 

14 

1 

& 

(34) 

2 

* 23 

(10) 

0 


''"No  Self-Paced  31E  data  regarding  EPIS  tests  were  available. 
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Table  16. 

Percentage  of 
Passing  EPIS 
Attempt 

CTS  and 
Tests  on 

Sel f-Paced 
the  First, 

Students  in  the  31 J Course 
Second  and  Third  (or  higher) 

CTS 

Sel f-Paced 

Task 

Attempt 

Percent 

N 

Percent 

N 

1 

1 

88 

(28) 

87 

(14) 

2 

9 

( 3) 

13 

( 2) 

3+ 

3 

( 1) 

0 

( 0) 

3 

1 

97 

(31) 

63 

(10) 

2 

3 

( 1) 

31 

( 5) 

3+ 

0 

( 0) 

6 

( 1) 

5 

1 

84 

(27) 

75 

(12) 

2 

18 

( 5) 

25 

( 4) 

3+ 

0 

( 0) 

0 

( 0) 

7 

1 

81 

(26) 

56 

( 9) 

2 

19 

( 6) 

31 

( 5) 

3+ 

0 

( 0) 

13 

( 2) 

15 

1 

100 

(32) 

100 

(16) 

2 

0 

( 0) 

0 

( 0) 

3+ 

0 

( 0) 

0 

( 0) 
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Table  17.  Percentage  of  CTS  and  Self-Paced  Students  in  the  35L  Course  Passing 
EPIS  Tests  on  the  First,  Second  and  Third  (or  higher)  Attempt 


CTS 

Self- 

Paced 

Task 

Attempt 

Percent 

N 

Percent 

N 

1 

1 

94 

(51) 

100 

(36) 

2 

6 

( 3) 

0 

( 0) 

3+ 

0 

( 0) 

0 

( 0) 

2 

1 

98 

(53) 

89 

(32) 

2 

2 

( 1) 

11 

( 4) 

3+ 

0 

( 0) 

0 

( 0)  l 

1 

3 

1 

96 

(52) 

100 

(36) 

2 

4 

( 2) 

0 

( 0) 

3+ 

0 

( 0) 

0 

( 0) 

4 

1 

100 

(54) 

72 

(26) 

2 

0 

( 0) 

28 

(10) 

3+ 

0 

( 0) 

0 

( 0) 

5 

1 

96 

(52) 

100 

(36) 

2 

4 

( 2) 

0 

( 0) 

3+ 

0 

( 0) 

0 

( 0) 
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students  and  instruction.  While  CTS  students  pass  most  Task  tests  on  the  first 
try,  thereby  showing  a high  level  of  subject  matter  mastery,  failure  does 
invoke  a computer-prescribed  remedial  sequence  which  must  be  completed  before 
the  student  is  permitted  another  try  at  the  test. 

The  fact  that  Task  Progression  Indices  under  CTS  tend  to  be  higher  than 
anticipated  may  be  partially  explained  by  the  individualized  remediation  capa- 
bility contained  within  the  CTS  instructional  model.  To  test  this  possibility, 
Rank-Order  correlations  were  computed  in  each  course  between  Task  Progression 
Indices  and  number  of  unsuccessful  attempts  at  Task  tests.  Should  this  relation 
hold  true,  we  would  expect  positive  correlations  between  these  two  variables. 

Tables  18,  19,  and  20  present  the  findings  for  each  of  the  courses.  For 
the  31E  course  (Table  18)  it  may  be  seen  that  in  all  Tasks  for  which  data  were 
available,  the  correlation  between  PI  and  number  of  "No  Go's"  is  positive  with 
the  exception  of  Task  nine.  For  eight  of  the  Tasks  as  well  as  for  all  Tasks 
taken  together,  the  correlations  are  significant  at  or  beyond  the  .05  confid- 
ence level.  Even  though  the  size  of  the  correlations  is  not  large,  their  dir- 
ection lends  support  to  the  notion  that  the  CTS  prescribed  remediation  require- 
ment as  a result  of  EPIS  test  failures  served  to  increase  the  Pi's. 

The  findings  for  the  31J  course  and  the  35L  course  also  tend  to  support 
the  suggested  influence  of  CTS  remediation  on  Task  Progression  Indices.  Table 
19  shows  that  of  the  four  31J  Tasks  for  which  data  were  available,  all  cor- 
relations for  two  Tasks  as  well  as  for  all  Tasks  considered  together  are  sig- 
nificant at  or  beyond  the  .05  level.  According  to  Table  20,  the  number  of 
"No  Go's"  in  the  35L  correlates  positively  with  Progression  Index  for  all 
Tasks  and  for  each  Task  taken  separately.  The  correlations  for  two  tasks  and 
for  all  Tasks  taken  together  are  significant  at  the  .05  level. 
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Table  18.  Rank  Order  Correlation  between  Task  PI  and  Number  of  Unsuccessful 
Attempts  at  Task  EPI5  Test;  for  CTS  Tasks  of  the  31 E Course 


Task 

r 

n 

1 

.17 

(46) 

2 

.27* 

(46) 

3 

.19 

(46) 

A 

.21* 

(46) 

5 

.38* 

(46) 

6 

• 

7 

.26* 

(46) 

8 

.17 

(44) 

9 

-.15 

(44) 

10 

.23* 

(45) 

11 

.16 

(44) 

12 

.39* 

(44) 

13 

.35* 

(44) 

14 

.21* 

(44) 

All  Tasks 

.31* 

- 

(AO) 

* ■ a <_  .05  one-tailed  (Kendall's  Tan) 

• “ not  computable 
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Table  19. 


Rank  Order  Correlation  between  Task  PI  and  Number  of  Unsuccessful 
Attempts  at  Task  EPIS  Test;  for  CTS  Tasks  of  the  31J  Course 


Task 

r 

n 

1 

.36* 

(32) 

3 

-.15 

(32) 

5 

.43* 

(32) 

7 

.06 

(32) 

15 

• 

All  Tasks 

.31* 

(32) 

* = a .05  one-tailed  (Kendall's  Tan) 

• « not  computable 
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Table  20.  Rank  Order  Correlations  between  Task  PI  and  Number  of  Unsuccessful 
Attempts  at  Task  EPIS  Test;  for  CTS  Tasks  of  the  35L  Course 


Task 

r 

n 

1 

.10 

(54) 

2 

.02 

(54) 

3 

.26* 

(54) 

4 

• 

5 

.23* 

(54) 

All  Tasks 

.24* 

(54) 

* “ a .05  one-tailed  (Kendall's  Tau) 

• ■ not  computable 
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5.  CTS  Costs 

As  part  of  the  CTS  Operational  Test  Plan  (September  1975),  a cost  model 
was  developed  that  was  to  provide  costing  information  for  management  and 
decision  making.  This  model  was  designed  for  a summative  evaluation  of  CTS 
costs  with  the  intent  of  generating  predictions  or  cost  estimates  of  a complete, 
fully  operational  500-terminal  system  (basically  CAI  in  nature).  No  attempt 
was  made  to  identify  costs  of  alternative  instructional  approaches  for 
comparative  purposes — this  was  to  be  considered  at  a later  date. 

At  the  time  our  evaluation  project  began,  cost  data  had  been  collected 
based  upon  the  Operational  Test  Plan  (OTP)  cost  model  (i.e.,  costs  were 
identified  and  categorized  as  described  in  this  model).  HumRRO  had  independently 
developed  a cost-effectiveness  specification  for  computer-based  training  systems 
(Seidel  and  Wagner,  1977).  We  intended  to  apply  this  specification  to  the 
CTS  data.  However,  it  was  not  possible,  in  most  cases,  for  the  cost  data  to 
be  organized  and  analyzed  within  the  framework  of  our  specification.  Rather, 
the  cost  categories  developed  by  CTS  (as  described  below)  were  used.  Werever 
possible,  guidelines  from  our  specification  (e.g.,  adjustments  for  inflation) 
were  adhered  to  in  calculating  CTS  costs. 

Cost  data  were  provided  by  CTS  personnel  for  FY  1973  through  FY  1977. 

As  presented,  the  data  were  organized  by  capital,  developmental,  and  operational 
costs  for  hardware,  software,  courseware  and  evaluation  activities. 

Development  costs  were  those  accrued  during  the  initial  development  of 
the  system. 

Capital  costs  included  all  money  spent  for  system  contracts,  course 
development,  programming  and  USASC&FG  personnel  time. 

Operational  costs  were  those  accrued  during  the  actual  operation 
of  the  system  (this  did  not  begin  until  FY  1977). 


109 


Personnel  assigned  to  develop  CTS  instructional  materials  were  required 
to  keep  daily  records  of  the  number  of  hours  spent  in  each  of  a number  of 
activities  associated  with  the  instructional  development  process.  We  were 
provided  with  copies  of  these  "time  logs"  for  all  personnel  for  the  years 
1974-1977.  These  time  data  were  converted  to  dollars  by  referring  to  the 
civilian  and  military  pay  scale  tables  appropriate  to  each  fiscal  year.  All 
recorded  costs  for  the  CTS  project  are  shown  in  Table  21.  All  contract  costs 
were  listed  in  the  Capital  Cost  category.  Fifty  percent  was  added  to  salaries 
to  cover  fringe  benefits,  indirect,  and  other  unrecorded  CTS-related  expendi- 
tures. This  50%  factor  was  used  to  weight  the  base  pay  of  civilian  and 
military  personnel.  It  was  based,  in  part,  on  guidance  from  the  Office  of  the 
Assistant  Secretary  of  Defense  (Comptroller) , "Economic  Cost  of  Military  and 
Civilian  Personnel,"  and  recent  policies  of  the  Office  of  Management  and 
Budget  regarding  civilian  personnel  pay.  The  following  percentages  of  base 
pay  were  used  in  computing  the  cost  of  civilian  personnel  services. 


Retirement  24.7% 
Health  Insurance  3.5% 
Life  Insurance  .5% 
Holidays  & Leave  16.5% 


45.2% 

The  remaining  4.8%  was  added  to  account  for  Government  contributions  to  other 
benefits,  and  to  cover  other  unrecorded  CTS-expenditures . To  adjust  costs 
for  inflation,  a rate  of  10%/year  (using  OSD  guidance)  was  multiplied  to  each 
prior  year's  costs  and  summed  over  all  years  to  arrive  at  the  estimated  total 
cost  of  CTS  (see  Table  21).  The  total  cost  of  CTS  by  the  end  of  FY  1977  is 
thus  estimated  at  $7.5  million  (in  1977  dollars). 
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Table  21.  Recorded  CTS  Costs 


Development 

Salaries 

FY73 

FY74 

FY75 

FY76 

FY7T/77 

TOTAL 

Hardware 

$ 34,508 

$ 39,006 

$ 25,202 

$ 36,546 

$ 10,558 

$ 145,620 

Sof tware 

34,754 

39,438 

25,202 

36,546 

21,641 

157,591 

Courseware 

151,378 

151,226 

24,262 

72,722 

19,161 

368,699 

Evaluation 

35,979 

34.436 

49,614 

51,042 

30,886 

701 ,957 

£ Rf»co rded  Developmental 


Salaries 

$ 

256j  579 

$ 

764,106 

$ 

124,780 

$ 

146,856 

$_ 

82,246 

$ 874,067 

Capital 

Salaries 

Hardware 

$ 

8,747 

$ 

35,453 

$ 

21,695 

$ 

28,257 

$ 

708 

$ 94,860 

Software 

8,747 

35,800 

21,695 

28,257 

20, 776 

115,275 

Courseware 

31E 

37,847. 

82,714 

85,389 

20,206 

226,1 51 

31 J 

37,842 

72,682 

99,941 

34,724 

245,189 

31L 

33,006 

71,112 

116,805 

27,712 

248,635 

Eval uat ion 

20,957 

60,021 

65,475 

146.453 

I Recorded  Capital  Salaries 

$ 

17,494 

$ 

200,900 

$ 

329,919 

$ 

424,124 

$ , 

104,126 

$1  ,076,563 

Contracts 

$ 

40,974 

$2,165,072 

$ 

120,875 

$_ 

424,498 

$_ 

240,130 

$2,991.545 

Operational 

Salaries 

Hardware 

$ 

19,799 

$ 19,799 

Software 

26,370 

26,370 

Courseware 

23,953 

23,953 

Evaluation 

— 

15,449 

15,449 

E Recorded  Operational  Salaries 

$_ 

85,571 

$ 85,571 

# 

ESTIMATED  TOTAL  CTS 

COSTS 

Estimated  Total  Costs 

FY73 

m± 

FY75 

FY7C 

FY7T/77 

TOTAL 

Contracts 

$ 40,974 

$2,165,072 

$ 

120,875 

$ 424,498 

$ 240,130 

$2,991,549 

Recorded  Salaries  (*  1.5)* 

411,110 

697.509 

681,299 

_ „8JL6jil°. 

407.915 

3,054,303 

Yearly 

452,084 

2,862,581 

802,174 

1,280,968 

648,045 

6,045.852 

Inflation  Factor 

(«  l-l)1* 

<»  1.1J3 

(«  i-n* 

(«  1.1) 

Total  CTS  Costs  (In  1977  dollars) 

S 661,896 

$3,810,095 

s= 

970,631 

$1  ,409  , 065 

$ 648,045 

$7  ,499,73? 

(end  of  E Y77 ] 

*50Z  added  to  Recorded  Salaries  for  Fringe,  indirect,  and  Other  Unrecorded  CTS-relatcd  expenditures . 
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13.  What  administrative  and  personnel  costs  were  incurred  to  establish 
in-service  training  programs? 


A question  of  concern  to  the  evaluation  effort  deals  with  the  administra- 
tive and  personnel  costs  incurred  to  establish  in-service  training  programs  for 
CTS  Instructors  and  for  instructional  programmers.  While  the  instructional 
programmers  were  not  privy  to  specific  costs,  nevertheless  they  were  asked  what 
classes  or  workshops  they  had  attended  in  preparation  for  their  jobs.  Nearly 
all  reported  that  they  attended  a one-week  workshop  on  Class  I given  by  the  Data 
Systems  Division  and  another  Class  I workshop  of  roughly  the  same  length  by  the 
Digital  Equipment  Company.  A number  reported  taking  a week-long,  in-house 
workshop  on  CAI  where  effective  writing  for  CAI  test  development,  and  branching 
principles  were  emphasized. 

Unfortunately,  the  time  logs  we  received  did  not  supply  complete  information 
describing  "training"  activities  in  FY75-FY77.  A total  of  1117  hours  of  person- 
nel time  (for  16  individuals)  was  listed  as  "in  training"  during  these  3+  years. 
We  are  sure  that  this  must  be  a partial  figure  and  thus  does  not  reflect  the 

total  cost  (in  time)  of  in-service  training.  We  also  did  not  receive  the 
costs  of  the  DEC  contract  for  its  on-site  training  program.  In  any  case,  based 
upon  our  knowledge  of  the  situation,  we  feel  that  a very  small  amount  of 
money  was  allocated  for  this  purpose.  As  is  pointed  out  elsewhere  in  this 
report,  a comprehensive  computer  "literacy"  program  was  necessary  for  assuring 
an  adequate  implementation  of  CTS.  Such  training  was  not  made  available  to 
all  concerned  individuals — though  it  should  have  been  and,  in  our  opinion, 
should  still  be  for  any  continued  CTS  operation. 


I 

I 

I 

I 

I 

I 
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Using  the  Chronological  Chart  of  Significant  Events  (Figure  1) 
provided  by  CTS  staff,  as  well  as  the  annual  reports,  the  acceptance  test 
reports,  and  the  CSSEA  evaluation,  a number  of  critical  management  decision 
points  relevant  to  implementation  are  highlighted  below.  It  can  be  stated 
at  the  outset  that  with  all  the  policy  changes  that  had  taken  place  during 
the  early  stages  of  this  effort,  there  was  ample  justification  for  delays  at 
the  beginning  of  the  project.  The  purpose  of  the  delays  would  have  been  to 
allow  for  alterations  in  the  implementation  plan  to  make  the  system  installa- 
tion and  development  effort  more  compatible  with  the  new  site,  different 
project  purposes,  and  new  personnel  at  USASC&FG.  The  criticality  of  these 
early  management  decisions  is  that  a dominoe  effect  was  initiated  and  has 
been  felt  subsequently  in  all  facets  of  the  project.  This  effect  prevents 
a meaningful  evaluation  of  the  system  as  a teaching  tool. 

Examples  of  the  times  at  which  management  decisions  were  made  that  affected 
the  course  of  the  CTS  project  are  as  follows: 

(1)  In  the  latter  part  of  April  1973,  it  was  known  that  the  Signal 
School  at  Ft.  Monmouth  was  to  be  closed.  On  4 May,  there  was  a Bidders'  Con- 
ference at  Ft.  Monmouth,  and  no  mention  was  made  of  this  change  in  location, 
nor  were  the  system's  performance  specifications  changed  to  accommodate  a shift 
to  a CMI  purpose.  Thirdly,  there  were  new  site  personnel  to  be  involved  and 
their  commitment  and  involvement  in  what  was  to  be  implemented  at  their  School 
should  have  been  recognized  and  obtained  at  that  time.  It  was  shortly  after 
this  critical  time  that  some  key  personnel  in  the  hardware/software  area;  e.g., 
Mr.  Allyn  Evans,  Supervisory  Computer  Systems  Analyst,  and  Mr.  John  Sintac, 
Computer  Systems  Analyst,  selected  other  federal  employment  at  Ft.  Monmouth 
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rather  than  move  with  the  project  to  the  anticipated  site  at  USASC&FG.  It 
should  be  noted  that  Mr.  Evans  was  the  key  person  in  setting  the  specifications 
for  the  CTS  system.  Mr.  Evans  did  return  to  the  CTS  project  some  6 months 
later  (August  1974  to  February  1975),  and  eventually  moved  with  the  PMO,  CTS 
to  Ft.  Eustis,  Virginia,  in  July  1975. 

(2)  It  was  not  until  16  July  1973,  that  three  proposals  were  received. 

At  this  time,  a delay  of  another  month  or  so  for  all  three  potential  con- 
tractors would  have  permitted  them  to  address  the  new  requirements. 

(3)  Another  critical  decision  point  occurred  with  the  initial  estab- 
lishment of  the  CTS  Field  Office  in  March  1974.  At  this  time,  there  should 
have  been  a computer  literacy  training  program  undertaken  with  the  various 
management  level  personnel  of  the  USASC&FG.  The  two  broadest  levels  of 
literacy — awareness  of  the  impact  of  computers  on  administration  and  knowledge 
of  the  effects  of  computers  on  the  various  curricula — should  have  been 
established  to  facilitate  adoption  and  cooperation  by  the  School  personnel. 

(4)  Another  critical  item  at  this  time  was  to  establish  within  the 
School  a single  chain  of  command  having  both  the  authority  and  responsibility 
for  the  system.  There  should  not  have  been  established,  as  noted  in  the 
Background  section  of  this  report,  a dual  and  relatively  ambiguous  chain  of 
command,  vis-a-vis,  the  project  structure  and  the  school  structure. 

With  the  help  of  Mr.  Frank  Giunti  and  Mr.  Donald  Kimberlin  of  the 
Communications  Technology  Office  at  Ft.  Eustis,  we  have  charted  the  structure 
of  the  CTS  Field  Office  responsibility  and  the  School  responsibility  as  it 
pertained  to  the  development  and  implementation  of  course  materials  within 
the  CTS  framework.  These  relationships  are  provided  in  Figure  2,  p.  18. 

This  dual  structur  led  to  administrative  ambiguity  over  course  development. 
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A temporary  resolution  of  course  development  responsibilities  occurred  when 
an  ADP  officer  took  a strong  position  on  course  development.  He  juggled  three 
courses  assigned  in  two  different  academic  departments  while  he  was  the  ADP 
officer  assigned  to  a third  non-academic,  but  support  department.  He  left  in 
mid-1976.  His  prime  duty  was  ADP  officer  for  the  USASC&FG.  There  was  no 
clear-cut  line  of  authority  in  USASC&FG  to  manage  course  development.  The 
nominal  task  leaders  were,  with  the  exception  of  the  present  one,  ADP 
specialists  with  GTS  course  development  responsibility  as  an  additional  duty. 
During  this  critical  period,  two  key  people  assigned  with  the  responsibility 
of  coordinating  the  USASC&FG  course  development  effort  were  each  given  a one- 
year  sabbatical  during  the  period  September  1976  to  December  1977,  a most 
critical  period  in  the  life  of  the  CTS  project. 

At  that  point,  two  critical  stages  in  the  CTS  project  had  already  been 
reached,  both  related  to  integration  of  CTS  as  part  of  the  ongoing  training 
effort  in  the  School.  Again,  as  noted  in  the  Background  section,  the  approach 
at  Ft.  Monmouth  was  oriented  toward  a laboratory  R&D  prototype  system  implemen- 
tation. However,  expectations  at  Ft.  Gordon  were  for  meaningful  cost- 
effective  results  that  reflected  the  nature  of  the  CTS  prototype  with  the 
ongoing  operational  requirements  of  USASC&FG. 

(6)  Throughout  the  history  of  this  implementation,  it  is  also  not  clear 
as  to  why  courses  were  chosen  that  were  previously  systems  engineered  and  self- 
paced,  thereby  already  having  resulted  in  a large  time  savings.  With  the 
cooperation,  control,  and  commitment  of  the  School,  a more  conducive  home  for 
a prototype  CMI  system  would  have  been  a lock-step  course  plagued  by 
instructional  deficiencies.  Ideally,  there  should  be  an  entire,  experimental 
site  in  which  to  test  out  any  prototype  computer-based  training  systems. 
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(7)  Another  item  critical  to  the  need  for  delay  and  re-evaluation  of  the 
entire  project  occurred  between  April  and  June  of  1975.  This  was  the  need  for 
a communications  study  to  be  accomplished.  It  required  a study  of  how  cabling 
might  be  installed  to  connect  the  different  buildings  in  which  the  terminals 
were  to  be  housed.  Again,  multiple  buildings  represented  a change  from  the 
original  plan.  It  would  have  been  perfectly  legitimate  and,  indeed,  desirable 
to  have  a delay  of  another  two  to  three  months  at  this  time  to  take  stock  and 
reassess  the  progre  s and  direction  of  CTS. 

(8)  In  the  software  area,  it  was  noted  on  the  Chronology  Chart  (Figure  1) 
that  the  designation  of  personnel  for  a CTS  Field  Office  occurred  in  August  of 
1974,  between  the  Phase  I and  Phase  II  acceptance  tests,  and  that  the  Field 
Office  was  not  fully  staffed  until  May  1975.  Inasmuch  as  the  applications 
programming  was  to  be  accomplished  at  the  CTS  Field  Office,  it  would  seem  that 
a delay  in  the  acceptance  test — the  second  phase  was  run  in  February  1975 — 
would  have  been  appropriate.^ 

(9)  There  were  a number  of  solid  attempts  to  record  costs  despite  all 
the  personnel  turbulence.  However,  the  individual  who  organized  .ihe  costing 
side  of  the  project  was  transferred  prior  to  the  completion  of  the  data  gather- 
ing activities.  The  result  was  that  information  was  not  gathered  as  compre- 
hensively and  completely  as  it  might  have  been. 

(10)  The  last  item  to  be  mentioned  in  the  way  of  critical  incidents  re- 
lates to  the  fact  that  the  first  students  were  started  on  the  CTS  system  (31E 
course)  in  January  1977.  But  from  the  data  gathered  by  HumRRO  in  the  struc- 
tured interviews  and  from  the  data  collected  from  the  system  printouts,  the 


See  page  6 of  the  third  year  status  report  regarding  systems  applications 
and  programming  staff  discussion,  Kimberlin,  1 August  1975. 
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software  was  still  not  stable  and,  therefore,  the  courses  should  not  have 


been  started  even  at  that  late  date.  If  there  is  a single  lesson  to  be  learned 
from  a prototype  implementation  involving  new  hardware  and  software,  as  well 
as  course  development — it  is  that  there  is  a serial  requirement  that  must  be 
met.  That  requirement  can  be  stated  as  follows:  The  hardware  anrl  software 
must  be  in  place  and  reliable  prior  to  any  attempt  to  run  students  on  the  system. 
Moreover,  the  development  of  course  materials  on  the  system  requires  a 
stability  in  the  interactive  relationships  needed  between  the  authors  and  the 
system.  This  period  of  criticality  occurred  in  January  1977  and  exists  until 
the  present  time. 

Suffice  it  to  say,  that  the  criticality  of  the  incidents  as  noted  above 
emphasized  the  need  for  delays  to  plan  the  approach  for  a proper  implementation 
of  CTS.  One  item  which  we  have  not  emphasized  but  which  should  be  noted  is  that 
the  preliminary  evaluation  plan  was  established  in  January  of  1974;  a consultants' 
conference  to  provide  comments  did  not  occur  until  almost  one  year  later, 

December  1974;  and  it  is  not  clear  that  the  consultants'  recommendations  were 
fully  implemented  because  of  the  tight  time  frame  of  the  entire  project.  If 
reasonable  delays  had  been  requested,  then  it  is  quite  possible  that  the 
consultants’  recommendations  would  have  been  implemented  in  a more  appropriate 
fashion,  and  that  the  operational  test  plan  would  have  reflected  changes  in 
the  system  and  facilitated  its  implementation. 
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12.  What  special  qualifications  are  required  by  the  instructional 
support  staff. 


Most  of  the  responses  to  this  question  in  our  interviews  were  centered 
on  instructional  programmers  and  IPES.  There  was  considerable  agreement  that 
the  qualifications  required  in  order  to  perform  instructional  programming 
include:  subject-matter  expertise,  knowledge  of  the  CLASS  I language,  knowing 

how  to  communicate  effectively  in  writing,  and  knowing  the  computer  system  and 
what  it  can  do  for  the  instructor  and  the  instructional  programmer.  In 
addition,  most  of  the  authors  agreed  that  knowledge  of  how  to  develop  CAI  and 
to  some  extent  self-paced  instruction  is  very  useful.  This  included  knowing 
how  to  employ  branching,  knowing  how  to  debug  instructional  materials,  how  to 
validate  them,  and  how  to  evaluate  them. 

Most  of  the  instrudtional  programmers  agreed  that  the  principal  require- 
ment for  IPES  is  knowing  how  to  type.  Two  said  they  have  to  know  the  CTS 
keyboard,  and  one  instructional  programmer  said  that  they  need  to  use  the 
editing  commands  in  order  to  enter  graphics.  Comments  on  computer  service 
personnel  were  very  limited.  The  only  substantive  qualification  stated  by  an 
instructional  programmer  is  that  the  computer  service  personnel  should  be 
familiar  with  the  courses  and  understand  the  problems  of  the  instructional 
programmers  and  the  instructors.  Another  remarked  that  a special  requirement 
for  computer  personnel  is  "patience." 
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14.  What  unanticipated  side  effects  or  by-products  can  be  attributed  to 
the  implementation  of  CTS? 


One  of  the  most  important  and  yet  often  overlooked  parts  of  evaluation, 
expecially  in  the  formative  sense  in  which  we  are  concerned  in  this  study  is 
the  unanticipated  results  that  can  be  attributed  to  the  prototype  system. 

In  the  current  implementation  there  were  two  side  effect  reports  generated. 

The  first  at  the  end  of  October  1976;  and  the  second  in  1977,  virtually  a 
year  apart.  It  is  important  to  note  that  the  first  side  effects  survey  was 
accomplished  at  a time  when  the  acceptance  of  the  system  had  just  taken  place. 

In  this  section  we  will  attempt  to  summarize  these  effects  in  narrative 
form,  and  to  note  changes  if  any  between  the  first  and  second  survey.  The 
individuals  supplying  the  information  for  this  survey  were  the  Chief  of  Data 
Systems  Division,  the  Director  of  the  Communications-Electronics  Department, 
the  Director  of  the  Specialized  Communication  Electronic  Equipment  Department 

of  the  USASC&FG  and  other  personnel  within  that  Denartment. 

There  was  an  apparent  lack  of  pre-  and  post-types  of  information 
from  the  same  personnel.  This  means  that  whatever  statements,  inferences, 
implications,  or  conclusions  can  be  drawn  from  the  side  effects  reports  must 
all  be  considered  highly  speculative  at  this  time.  Unanticipated  effects 
can  be  either  beneficial  or  negative  with  respect  to  innovation.  The 
questionnaire  as  distributed  by  CTS  personnel  stressed  negative  effects  and 
therefore  some  benefits  which  might  have  been  discussed  by  the  respondents  were 
excluded . 

The  comments  in  the  initial  1976  reports  suggest  a need  for  better  manage- 
ment of  the  CTS  Implementation.  They  suggest  a single  line  of  command  and  the 
need  for  more  adequate  definition  of  responsibility  within  such  a project. 
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Other  comments  in  the  1976  reports  Indicated  a problem  in  instability 
of  course  materials  and  of  hardware  and  software.  It  appears  that  none  of 
the  individuals  surveyed  expected  a prototype  implementation.  Rather,  they 
were  expecting  an  operational  curriculum  along  with  operationally  stable 
hardware  and  software.  In  fact,  many  of  their  suggestions  for  improvements 
relate  to  just  these  points. 

A concern  in  both  administrations  of  this  survey  was  for  greater  training 
and  familiarization  both  at  the  instructional  programmer  level  and  at  the 
management  level  for  all  persons  who  might  be  involved  in  such  an  implementa- 
tion. These  suggestions  also  support  the  previously  discussed  implementation 
issue  of  orientation  and  technical  literacy  training  with  respect  to  the  CTS 
computer  system.  A point  related  to  this  need  for  computer  literacy  is  clearly 
indicated  in  one  of  the  respondent's  forms.  This  individual  felt  that  CTS 
will  continue  to  have  adverse  effects  such  as  "increasing  the  training  time  for 
each  student,"  and  "increasing  the  course  failure  rate."  The  training  litera- 
ture on  CAI  and  self-pacing  clearly  yields  opposite  conclusions  to  this  state- 
ment. The  point  here  is  that  the  individual  respondent  is  making  an  incorrect 
inference  based  upon  a lack  of  adequate  orientation  and  pre-training  in 
areas  of  CAI,  CMI,  and  individualization  of  instruction.  In  addition,  this 
individual  is  also  reflecting  a lack  of  understanding  of  the  difference 
between  an  operational  course  already  debugged,  formatively  evaluated  and 
stable,  as  opposed  to  a prototype  computer  based  training  system  that  was  not 
stable  during  its  period  of  implementation. 

The  1977  survey  essentially  supports  the  same  notion  of  a premature 
implementation  based  upon  operational  expectations  of  the  users.  In  addition, 
the  fact  that  different  individuals  have  filled  out  the  questionnaires  from 
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year  to  year  emphasizes  the  problem  of  personnel  turbulence.  What  is  sti- 
pulated is  a need  for  course  development  and  other  CTS-related  personnel  to 
be  trained  and  retained  In  the  same  positions  until  the  system  is  stabilized 
to  make  certain  that  the  project  is  successful. 

The  instability  of  the  course  structure  and  POIs  caused  confusion  in  the 
minds  of  the  personnel  who  developed  course  materials.  They  wished  to  keep 
such  changes  to  a minimum  until  the  course  had  been  fully  implemented.  In 
any  operational  system  this  clearly  would  be  required. 

There  were  also  some  technical  suggestions  made,  such  as  the  need  for 
some  means  of  projecting  a picture  or  secondary  visual  at  the  student  termi- 
nal. That  is,  the  need  for  graphics  devices  was  highlighted.  Also,  the 
need  for  improved  disk  allocation  or  disk  space  was  mentioned.  It  is  import- 
ant to  note  that  such  changes  could  have  been  made.  The  point  here  is  that 
some  of  these  technical  decisions  could  have  been  made  had  the  proper  management 
been  available  during  the  course  of  the  project.  There  was  no  one  responsible 
for  insuring  that  timely  changes  were  made  as  required  within  the  project. 

Thus,  some  of  the  technical  problems  could  have  been  solved  with  the  current 
hardware  had  the  authority  and  responsibility  been  invested  in  a single 
individual . 
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E.  CONCLUSIONS  AND  RECOMMENDATIONS 


This  section  of  the  report  is  organized  into  three  major  sub-sections. 
First,  we  will  summarize  our  evaluation  findings — by  major  area  of  concern  and 
within  each  area,  by  the  relevant  evaluation  questions  we  sought  to  answer. 
Second,  we  will  discuss  the  possibility  of  cost-effectiveness  projections  for 
CTS  if  it  were  to  be  used  as  a CMI  system  devoted  to  handling  all  the  adminis- 
trative needs  of  the  USASC&FG.  Finally,  we  will  present  a section  in  which  we 
describe  the  lessons  that  were  learned  from  the  CTS  project  along  with  guidance 
for  those  who  wish  to  Implement  computer-based  or  any  other  innovative 


training  systems. 


1 . Summary  of  Evaluation  Findings 


In  this  section  we  will  summarize  our  findings  for  each  of  the  six 
evaluation  areas  covered  in  our  study.  Within  each  major  area  of  concern, 
the  relevant  evaluation  questions  will  be  briefly  addressed. 

Technical  Effectiveness 

The  CTS  hardware/software  configuration  of  six  minicomputers  with  128  CRT 
terminals  was  not  designed  to  optimally  meet  requirements  for  computer-managed 
instruction  at  Ft.  Gordon,  or  other  Army  training  installations.  Some  key 
instructional  management  functions  are  not  being  supported  even  though  the 
current  hardware  is  underutilized.  One  or  more  of  the  DC's  could  be  modified 
to  directly  support  CMI  functions.  Once  such  changes  are  made,  an  analysis 
of  the  CMI  utility  to  the  whole  school  of  CTS  could  be  performed. 

CTS  Course  Development 

Instructional  materials  and  logic  development  for  CTS  follows  a systematic 
iterative  process  which  begins  with  document  review,  selection  of  subject  matter 
for  on-  or  off-line  presentation,  and  flowcharting  student/system  interaction. 
Having  developed  these  detailed  specifications,  the  Instructional  Programmer  (IP) 
prepares  the  actual  on  and  off-line  content  and  logic.  Then,  either  personally 
or  by  delegation  to  a specially  trained  typist,  enters  the  materials  into  the 
CTS  computer.  Subsequent  steps  in  the  development  process  entail  review,  de- 
bugging, and  revision  based  on  (1)  evaluation  by  other  instructors  and  IP's, 
and  (2)  the  on-line,  interactive  experiences  of  advanced  and  naive  students. 

Once  the  material  has  been  "shaken  down"  it  is  incorporated  with  ongoing 
instruction. 
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How  much  time  is  required  to  prepare  the  training  and  test 
materials  contained  in  the  operational  self-paced  course 
annexes  to  CTS  instructional  materials ? 


On-line  materials  accounted  for  about  11.7%  of  the  POI  hours  contained 
In  the  three  courses.  However,  of  the  55,000  hours  spent  in  materials  develop- 
ment, approximately  65%  was  devoted  to  preparation  of  on-line  materials. 
According  to  IP's  Interviewed,  on-line  development  activities  consisted  of 
original  authoring,  review,  debug,  and  revision  rather  than  conversion  or 
modification  of  existing  self-paced  materials. 

What  feedback  is  available  for  you  to  use  in  revising 
instructional  materials  and  tests?  How  has  it  been  used? 

What  additional  feedback  is  necessary?  How  would  it 
be  used? 

The  principal  sources  of  feedback  were: 

• student  comments  in  the  classroom 

• IP  observation  of  students  and  instructurs 

• review  of  their  materials  by  other  IP's. 

Very  few  IP's  reported  making  consistant  use  of  either  CTS-generated 
reports  or  manual  records.  This  is  related  to  the  finding  discussed  earlier 

on  page  51  that  only  20%  knew  about  the  five  CTS  reports. 

For  the  most  part  IP's  were  not  able  to  articulate  a systematic  procedure 
for  identifying  specific  problem  areas  in  the  instruction  or  for  correcting  ma- 
terial based  on  feedback.  Suggestions  for  additional  types  of  feedback  were 
virtually  absent. 

What  is  the  average  development  time  (hours)  required 
for  one  POI  hour  of  instruction  in  the  CAI/CMI  mode? 

The  average  number  of  course  development  hours  per  hour  of  on-line  in- 
struction for  all  courses  was  175;  the  range  was  144:1  (35L)  to  197:1  (31J) . 
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What  difficulties  have  been  encountered  in  fitting  the 
previously  developed  self-paced  instructional  materials 
into  the  CTS  instructional  model? 

Instructional  Programmers  reported  that  little  difficulty  was  experienced 
in  coordinating  the  existing  self-paced  materials.  This  is  to  be  expected  be- 
cause the  older  materials  were  retained  in  an  off-line  format.  Required  on- 
line materials  were  developed  from  scratch.  The  minor  problems  which  did 
occur  resulted  from  detailed  work  associated  with  ensuring  the  smooth  coordi- 
nation of  on-  and  off-line  instruction. 

To  what  extent  does  the  system  enable  timely  modification 
(e.g.3  due  to  POI  changes),  revision  and  validation  of 
course  materials? 

IP's  believe  that  the  CTS  system  enhances  the  effectiveness  and  efficiency 
of  the  validation  of  instruction.  Opinions  concerning  the  value  of  CTS  in 
support  of  materials  revision  were  divided;  about  half  reported  difficulty 
in  making  revisions.  Finally,  the  majority  of  IP's  did  not  believe  that  the 
CTS  review/revision  process  reduced  the  time  required  to  introduce  materials 
into  the  classroom. 

What  special  problems , if  any,  were  encountered  when  entering 
( inputting ) training  materials  on-line? 

Frequent  or  intense  problems  associated  with  inputting  training  materials 
included: 

• inability  of  Instructional  Program  Entry  Specialists  (IPES) — specially 
trained  clerk/ typists — to  work  effectively  and  without  supervision 

• inadequate  IP  training  in  the  CLASS  I language1 

• a number  of  acute  system  hardware/software  problems  resulting  in 
loss  of  stored  materials,  slow-down  of  the  debug-revision  process, 
and  delays  in  course  implementation. 


^EDITOR  commands,  a subset  of  CLASS  I,  were  used  to  enter  the  materials. 
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CTS  Course  Administration 


An  extremely  Important  function  of  the  CTS  is  to  relieve  the  instructor 
of  many  administrative  and  record  keeping  responsibi lites  so  that  he  can 
concentrate  his  efforts  on  providing  precise,  timely  attention  to  unique 
student  learning  problems.  To  this  end  the  designers  of  CTS  incorporated 
system  generated  reports,  system  monitoring  of  routinized  student  progression 
through  the  instruction,  and  maintenance/monitoring  of  instructional  resources 
to  optimize  their  availability  in  the  classroom. 

Below  we  summarize  findings  on  the  success  of  the  computer-based  adminis- 
trative support. 

Hew  useful  are  the  CTS-generated  reports  for  instructors 
and  training  managers? 

Six  CTS  reports  provide  information  on  the  degree  to  which  students,  as 
a group,  have  been  successful  in  mastering  course  content;  these  include  the 
Course/Task  Report,  the  TAIS  Report,  and  three  reports  presenting  data  on  EPIS 
and  pre/posttests.  Twenty  percent  or  fewer  of  instructional  personnel  can- 
vassed were  aware  of  these  reports;  however,  those  few  who  were  familiar  with 
them  regarded  the  forms  positively.  Most  personnel  were  familiar  with  reports 
related  to  student  scheduling  and  throughput  (e.g.,  Class  Roster,  Student 
Activity  Report,  and  Graduation  Prediction).  For  the  most  part  opinions  con- 
cerning the  accuracy  and  utility  of  this  class  of  reports  were  favorable. 

Has  the  system  been  effective  in  monitoring  students 
within  the  respective  flasks/ Annexes? 

Instructors  report  that  under  "normal"  student  flow  the  system  can  route 
the  student  from  Task  to  Task  through  the  course  and  indicate  his  position  with 
few  problems  or  errors.  The  system  is  considered  less  capable  of  routing 
students  when  the  demand  for  equipment  and  practice  stations  becomes  intense. 
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Instructor  opinion  Is  divided  on  whether  CTS  reduces  their  responsibility  and 
workload  in  the  areas  of  routing  students  and  keeping  related  records. 

Were  sufficient  resources  available  to  handle  the  student 
load ? 

From  all  available  evidence,  it  appears  that  CTS  resources  were  able  to 
handle  student  throughput.  No  untoward  queueing  occurred  at  computer  terminals 
and  student  interactions  took  place  when  required. 

Is  there  an  adequate  back-up  capability  to  provide  instruction 
during  computer  down-time? 

As  a result  of  our  observations  and  interviews,  we  believe  that  on-line 
instruction  comprised  less  than  12%  of  the  courses  and,  as  such,  instructors 
were  not  overtaxed  when  CTS  was  inoperative.  Available  back-up  materials  were 
sufficient  to  ensure  the  continuation  of  instruction  during  computer  down- 
time. 

Training  Effectiveness 

What  is  the  average  student  progression  index  by  Task  and 
Annex  for  each  CTS  course?  What  is  the  standard  deviation? 

How  many  graduates  were  there  in  each  course? 

The  average  student  progression  index  for  CTS  students  in  the  31E  Course 
was  1.11  while  that  for  self-paced  students  was  .84.  For  the  31J  Course  the 
average  PI  among  CTS  students  was  1.01;  the  average  for  self-paced  students 
ranged  from  .98  to  1.08  depending  on  the  POI  in  effect.  The  average  PI  for 
both  CTS  and  self-paced  students  in  the  35L  Course  was  .96. 

The  31E  Course  graduated  205  CTS  students  during  FY  1977  and  97  more  in  the 
period  October  1977  - March  1978.  In  the  35L  Course,  70  CTS  students  graduated 
in  the  period  July  1977  - May  1978.  Eight  students  have  graduated  the  31J 
Course  since  its  CTS  tasks  were  declared  operational. 
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What  was  the  number  and  percentage  of  students  who 
fabled  to  graduate  from  each  CTS  course? 

In  FY  1977  the  31E  course  lost  22  students  for  academic  reasons;  the  same 
number  were  lost  during  October  1977  to  March  1978;  total  losses  are  about  11%. 

In  the  35L  Course  eleven  students  failed  to  graduate  during  the  period  July  1977- 
May  1978,  around  12  percent. 

What  are  student  attitudes  towards  CTS  courses? 

Student  responses  to  a survey  on  CTS  (see  pp.  97-98)  revealed  that  the 
majority  found  more  to  dislike  about  this  instructional  mode  than  to  like. 
Students  believed  the  instruction  too  hard,  the  instructors  less  available 
than  they  should  be,  and  the  variety  of  media  difficult  to  work  with.  They 
also  considered  system  down-time  and  shortage  of  terminals  an  impediment  to 
learning.  About  two-thirds  said  they  would  not  prefer  more  instruction  delivered 
via  CTS. 

What  is  the  relationship  between  student  scores  on  CTS 
administered  Task  tests  and  Progression  Index  values? 

What  percentage  of  students  passed  CTS  tests  on  the 
first  try? 

Available  data  indicate  that  CTS  students  pass  Task  tests  on  the  first 
attempt  somewhat  more  frequently  than  their  self-paced  counterparts.  In  the 
31J  Course  CTS  students  experienced  a considerably  higher  first-try  success 
rate  in  the  majority  of  comparable  tasks.  Success  on  the  first  attempt  at 
Task  tests  in  the  35L  Course  was  high  for  both  CTS  and  Self-Paced  students 
with  the  exception  of  one  Task  where  100%  of  CTS  students  passed  on  the  first 
try  as  opposed  to  72%  of  Self-Paced  students.  Comparisons  were  not  possible 
for  the  31E  Course  because  no  data  was  available  for  Self-Paced  students;  none- 
theless at  least  80%  of  CTS  students  passed  the  majority  of  31E  task  tests  on 
the  first  attempt. 


129 


Statistical  analysis  indicates  significant  positive  correlations 
between  number  of  Task  test  "NO  GO' s"  and  size  of  Progression  Index  for 
each  CTS  course.  It  is  suggested  that  required  remedial  sequences  following 
Task  Test  failure  in  CTS  courses  may  account  for  the  relationship. 

CTS  Costs 

Costs  and  personnel  time  data  provided  to  us  by  CTS  personnel  were  organized 
into  Development,  Capital,  and  Operational  categories  established  by  their 
Operational  Test  Plan  (OTP)  cost  model.  Total  costs  of  CTS  for  the  period 
FY  1973-FY  1977  (in  1977  dollars)  was  estimated  at  $7.5  million. 

What  administrative  and  personnel  costs  were  incurred 
to  establish  in-service  training  programs? 

From  only  partial  data,  we  estimate  that  approximately  1100  hours  (16 
individuals)  were  accounted  for  by  in-service  training  programs.  However, 
we  feel  that  one  should  attach  considerable  importance  and  resources  to  such 
training  in  order  for  an  implementation  such  as  CTS  to  be  successful. 


Implementation  Issues 

Management  decisions  made  early  in  the  CTS  project  have  affected  its 
course  until  now.  Some  of  the  decisions  that  are  critical  to  a successful 
implementation  are  listed  below: 

• Once  the  change  in  site  and  purpose  becameknown,  the  CTS  project 
should  have  been  delayed  to  permit  alterations  in  system 
performance  specifications. 
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• Once  the  site  shifted,  a computer  literacy  training  program 
should  have  been  undertaken  for  USASC&FG  personnel  at  all 
levels. 

• A single,  not  dual  chain  of  command  should  have  been  established 
within  the  USASC&FG  that  would  have  had  the  authority  and 
responsibility  for  the  system. 

• Due  to  the  hardware/software  instability  even  as  late  as  January 
1977,  no  on-line  course  development  or  actual  student  interactions 
should  have  begun  even  then.  A serial  development  of  CTS  should 
have  been  required,  even  if  delays  had  to  be  requested. 

What  special  qualifications  are  required  by  the  instructional 
support  staff? 

Respondents  agreed  that  data  entry  specialists  had  to  know  how  to  type. 
Instructional  programmers  required  subject-matter  expertise,  knowledge  of  the 
CLASS  I language,  writing  skills,  and  computer  literacy. 

What  unanticipated  side  effects  or  by-products  can 
be  attributed  to  the  implementation  of  CTS ? 

Only  the  problems  or  negative  effects  were  obtained  bv  the  survey — namely 

- lack  of  clear  management 

- instability  of  hardware/software  and  course  materials 

- lack  of  adequate  training  for  instructional  personnel 
related  to  computer' based  training 

- personnel  turbulence 
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2.  Cost  Effectiveness 


Projections 

As  part  of  our  study  goals,  we  were  to  perform  a predictive  cost  analysis 
of  CTS  assuming  the  system  was  to  be  replicated.  However,  insufficient  opera- 
tional cost  data  were  available  to  make  any  valid  cost  projections.  Also,  no 
direct  comparison  between  CTS  and  t no  self-paced  courses  was  possible  as  costs 
for  the  latter  were  not  calculated. 

Regarding  such  cost-effectiveness  ratios,  any  projections  calculated  dur- 
ing the  initial  development  stages  of  the  training  system  should  be  treated  with 
caution  as  they  are  subject  to  much  error.  This  is  especially  true  when  a pro- 
totype hardware/software  system  is  undergoing  modification.  One  should  adopt 
an  approach  that  will  accumulate  cost-effectiveness  data  on  a continuing  basis 
throughout  the  life  cycle  of  a project.  Data  collected  during  the  Development 
Phase  should  be  used  in  a formative  manner — for  system  revision  and  improvement. 
Data  collected  during  its  Operational  Phase  could  then  be  used  to  provide  more 
reasonable  and  valid  projections  for  decision-making. 

A cost-effectiveness  analysis  should  be  performed  on  training  systems  when 
they  are  fully  operational.  However,  as  our  study  of  GTS  indicates,  one  of  the 
three  courses  on  the  system  was  still  not  operational  by  the  time  our  data  were 
collected.  Thus,  CTS  was  in  a Development  Phase  at  the  time  cost  data  were  col- 
lected. Projections  based  on  these  data  must  be  considered  hypothetical.  The 
cos ts  and  effectiveness  data  that  were  gathered  while  CTS  was  under  development 
were  influenced  by  many  factors  that  would  not  occur  in  a more  stable  environment. 
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A comparison  of  alternative  training  systems  on  the  basis  of  coats  and/or 
effectiveness  Is  meaningful  only  if  the  systems  contain  courses  with  similar 
objectives,  content,  testing  conditions  and  criteria.  The  self-paced  and  (ITS 
versions  of  the  31E,  31J  and  35L  courses  do  not  meet  all  these  comparability 
criteria,  and  thus  the  comparisons  made  in  thl.c  report  are  tenuous  at  best. 

it  is  inappropriate  to  try  to  retrofit  a cost-effectiveness  model  and  sub- 
sequent analyses  on  data  collected  when  Implementing  another  model  that  was 
designed  to  meet  another  purpose.  The  cost  model  prepared  in  the  (ITS  Opera- 
tional Training  Plan  (OTP),  September  1975,  was  designed  for  evaluating  the 
cost-effectiveness  of  a CAI  system.  The  cost-effectiveness  ratios  identified 
in  that  document  such  as  Hourly  Instructional  Cost  are  valid  for  that  use  of 
the  computer.  However,  once  a CMI  purpose  was  imposed  on  the  system — a different 
cost-effectiveness  index  would  be  more  useful  [e.g.,  graduation  cost  (cost  per 
student)].  In  either  case,  CTS  would  not  be  cost-effective  unless  the  personnel 
who  are  to  be  replaced  by  the  labor-saving  assistance  of  the  system  are,  in  fact, 
replaced.  If  not,  then  no  large  system  such  as  this  can  ever  be  judged  cost- 
effect  ive . 

The  total  CTS  costs  described  in  the  previous  section  of  approximately 
$7.5  million  would  not  need  to  be  duplicated  if  another  system  with  identical 
specifications,  software  and  courseware,  were  to  be  dove  1 oped/purchased . How- 
ever, CTS  personnel  have  indicated  that  they  would  redesign  the  configuration 
with  less  expensive  hardware/terminal s . Approximately  $1.5  million  was  estimated 
to  be  needed  to  replicate  the  CTS  hardware/software  system.  In  addition,  a 
courseware  development/evaluation  effort  would  be  needed  (estimated  at  $1 . 85 
mill  Ion  for  3 courses) . Such  high  investments  would  need  to  be  amortized  over 
the  useful  life  of  the  system  (8  y*ars  for  the  hardware/sof tware ; U years  for 
the  courseware).  This  means  that  approximately  $650,000  of  amortized  cost 
needs  to  be  accounted  for  each  year. 
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Other  operational  costs  would  include  the  salaries  of  those  personnel 
required  to  operate  the  CTS.  Such  personnel  would  include  a Systems  Analyst 
(First  Lieutenant),  a Programmer  (E6),  an  Administrator  (GS-12)  serving  as 
an  interface  with  USASC&FG  personnel,  a Console  Operator  (LA),  and  part  (1/3) 
of  a Shift  Supervisor's  (GS-7)  time.  The  yearly  salaries  of  this  team  would 
be  approximately  $10A,315  (fully  burdened  with  a 50%  factor).  The  number  of 
personnel  who  would  be  replaced  by  the  system  to  handle  3 courses  would  only 
account  for  $72,000  or  less  than  70%  of  the  additional  cost  to  operate  CTS. 

It  may  be  that  if  CTS  was  to  be  used  in  a CMI  mode  exclusively — no  on-line 
instructional  interactions — then  administering  the  schedules  of  6000-8000 
students  could  be  economically  Justified.  The  COMTRAINS  report  (Appli-Mation, 
1978)  predicted  that  the  current  CTS  system  could  handle  an  on-board  training 
load  of  6000-8000  students  at  USASC&FG, if  the  system  was  to  be  used  entirely 
in  a CMI  mode.  Such  a use  has  not  been  evaluated,  nor  have  the  necessary 
additional  courseware/software  costs  been  calculated. 

In  making  cost-effectiveness  projections,  we  need  to  consider  all  appro- 
priate costs  and  benefits  during  the  operational  life  of  the  system.  Thus, 
the  total  costs  that  may  be  avoided  during  this  period  (e.g.,  additional 
personnel  to  handle  an  alternative  system)  need  to  be  considered  in  Judging 
the  potential  value  of  the  CTS. 

The  numbers  in  Table  22  reflect  the  maximum  load  that  npeds  to  be  managed 
by  CTS  in  a purely  CMI  mode  (i.e.,  no  instructional  Interactions  or  testing). 

It  was  not  the  focus  of  this  project  to  analyze  the  cost-effectiveness  of  such 
an  application,  but  to  point  out  that  it  has  been  estimated  that  only  $72,000/ 
year  in  administrative  costs  could  be  saved  (personnel  salaries)  by  CMI  imple- 
mentation of  CTS  in  the  three  courses  discussed.  However,  we  do  not  know  the 
total  level  of  personnel  reduction  if  all  of  the  courses  at  Ft.  Gordon  were  to 


be  self-paced  and  required  manual  scheduling  and  management.  The  Issue  here  Is 
one  of  cost  avoidance,  as  It  Is  highly  probable  that  additional  personnel  would 
have  to  be  hired  to  do  this  work.  A careful  analysis  needs  to  be  done  to  de- 
termine the  additional  software  and/or  hardware  necessary  to  handle  all  Ft. 
Cordon' 8 self-paced  course  management  by  CTS  and  whether  this  extra  expense 
is  less  than  that  of  a comparable  manual  operation.  Only  then  can  this  issue 
be  resolved. 


Table  22.  Estimated  Input  and  Training  Load  for  Ft.  Gordon 


FY78 

FY79 

FY80 

Total  Input 

Ft.  Gordon  29,426 

Average 
Training  Load 
6288 

Input 

29,502 

Average 
Training  Load 
8814 

Input 

33,025 

FY81 


Input 

32,133 
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3.  Lessons  Learned/Guidance 


la  CTS  a Technically  Effective  CMI  System? 

One  of  the  most  frequently  asked  questions  of  us  as  evaluators  has  been: 
"Is  the  CTS  hardware/software  configuration  adequate  for  the  Army's  CMI  needs 
either  at  Ft.  Gordon  or  at  other  Army  installations?"  It  is  the  case  that 
CTS  probably  lias  sufficient  CMI  capacity  for  handling  the  scheduling,  adminis- 
tration and  record  keeping  of  all  the  students  in  a self-paced  mode  at  Ft. 
Gordon.  However,  we  did  not  investigate  this  matter  directly  but  must  rely 
on  the  CSSEA  (1977)  and  COMTRAINS  (Appli-Mation,  1978)  reports  for  their  con- 
clusions. Both  reports  suggest  that  with  some  modifications  to  the  software 
and  with  additional  hardware  links,  the  current  system  could  provide  sufficient 
capability  for  all  of  Ft.  Gordon's  CMI  needs. 

In  our  study  we  learned  that  the  system  did  not  provide  the  kind  of  re- 
liable feedback  that  would  give  the  instructors  confidence  to  use  the  system. 
However,  much  of  what  happened  was  a result  of  the  inherent  instabilities  of 
a prototype  system,  management  ambiguity,  and  administrative  difficulties. 

For  example,  on  the  technical  side,  one  of  the  problems  that  people  using  the 
system  complained  about  was  that  only  one  course  could  be  on  the  system  at  any 
one  time  for  authoring  purposes.  The  fact  of  the  matter  is  that  this  occurred 
because  there  was  simply  not  enough  disk  space  allocated  to  instructional 
development.  While  there  was  limited  disk  space  for  development  due  to  the 
systems  design,  there  was  management  of  the  available  resources.  In  the 
development  process,  USASC&FG  was  faced  with  two  alternative  procedures  for 
allocating  disk  space  and  system  time.  One  was  to  give  each  course  a small 
amount  of  disk  space,  but  allow  all  courses  on-line  at  the  same  time.  The 
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I other  waa  to  aLlow  one  course  at  a time  on  the  system,  but  give  them  more 

work  area  on  the  disk.  Both  alternatives  were  used  during  the  development 
process.  Early  in  the  process  all  courses  were  allowed  to  work  on-line  at  the 
same  time  because  an  IP  needed  to  work  longer  on  one  lesson  unit.  As  develop- 
ment progressed,  the  other  procedure  was  used  because  the  IP  needed  access  to 
many  units  for  minor  revisions. 

Also,  there  was  no  one  management  group  or  manager  responsible  for  the 
GTS  training  development  within  the  School  itself.  Such  management  responsi- 
bility should  be  vested  in  one  individual  in  the  office  of  the  Deputy  Commandant 
who  would,  in  turn,  be  responsible  to  the  Commandant  of  the  School . That 
person  would  then  have  authority  over  the  various  support  facilities,  including 
the  Data  Systems  Division,  as  well  as  the  Departments  and  instructor  personnel. 

Would  a CTS  CM  I System  be  Cost  Effective? 

Whether  or  not  CTS  is  or  would  be  cost-effective  will  depend  on  the  al- 
ternative manual  system  against  which  it  is  compared.  For  example,  could  on- 
board personnel  handle  the  planned  administrative  requirements  for  self-pacing 
at  Ft.  Cordon?  If  so,  how  many  current  personnel  would  be  replaced  by  the 
computer?  If  not,  how  many  additional  personnel  would  be  required?  The  cost- 
effectiveness  conclusion  would  also  depend  on  the  replacement  of  current 
terminals  in  the  CTS  configuration  by  less  expensive,  more  suitable  equipment 
for  CMI  purposes.  The  requirements  for  such  minimal  hardware  and  software 
changes  nevertheless  must  be  costed  in  order  to  determine  if  CTS  can  be  an 
efficient  system  for  CMI  use  at  Ft.  Gordon.  The  generalizability  of  this  type 
( of  system  for  other  installations  is  not  recommended  without  a detailed  analysis 

of  the  CMI  needs  at  other  installations  and  a comparison  with  other  possible 
systems.  With  minimal  CMI  requirements,  much  less  expensive  system  equipment 
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can  be  obtained.  While  this  equipment  may  have  less  software  capabilities  they 
could  be  more  cost-effective  (MI  systems  depending  upon  the  specific  needs 
at  other  sites. 

The  following  paragraphs  describe  the  "lessons  learned"  in  our  evaluation 
of  CTS  and  presents  point  to  consider  when  planning  an  innovative  training 
system  implementation. 

• Ensure  that  there  is  universal  agreement  (or  understanding)  on  project 
purpose (s) . 

At  the  very  beginning  of  the  development  of  a conceptual  plan  the  state- 
ments of  project  purpose  will  set  the  stage  for  expectations  of  potential  out- 
comes. In  this  respect,  the  CTS  project  was  originally  to  be  considered  as  a 
prototype  CAX  system  implementation.  The  staff  and  user  expectations  were 
established  at  Ft.  Monmouth  where  the  system  was  to  be  implemented  in  prototype 
form.  The  participants  at  that  school  were  familiar  with  CAI  ns  a result  of 
their  earlier  studies.  Moreover,  their  expectations  were  in  fair  agreement  as 
to  the  potential  outcomes.  However,  when  the  system  was  moved  to  Ft.  Gordon 
a reestablishment  of  agreement  on  purpose  should  have  been  accomplished  with  new 
user  and  new  staff  personnel  there.  In  the  future  many  of  the  difficulties  that 
have  been  documented  in  this  report  cam  be  ameliorated,  if  not  totally  eliminated, 
by  consistent  coherent  planning  and  agreement  on  purpose. 

• Make  certain  that  the  system  design  is  compatible  with  the 
project  purpose(s). 

The  requirements  for  a CAI  system  are  different  than  the  requirements  for 
a CMI  hardware /sof tware  system.  For  example,  a CMT  terminal  may  require 
compatabl lity  with  optical  scanning  equipment.  The  power  and  configuration  of 
the  computer  system  will  also  differ  depending  upon  whether  it  is  a CAI  or 
CMI  system.  In  the  CTS  project,  the  hardware/software  design  was  originally 
stipulated  to  be  a CAI  system  with  a certain  percentage  of  graphics  and  a 
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large  percentage  of  on-line  interactive  instruction.  These  requirements  were 

totally  changed  when  the  Implementation  at  Ft.  Gordon  was  initiated.  The  purpose 

became  that  of  CMI  and  therefore  the  hardware/software  system  should  have  been 

reconfigured  to  handle  the  new  requirements  and  new  purposes.  In  the  future, 

management  decision-making  must  be  flexible  enough  to  present  hardware  and 

software  requirement  modifications  to  handle  the  shift  in  purposes. 

• Ensure  that  there  is  a serial  development  of  critical 
system  components. 

One  should  sequentially  organize  the  system  development  activities  within 
a project  of  this  type.  Unlike  the  parallel  system  development  events  encoun- 
tered at  CTS,  the  following  sequence  for  training  system  development  should  be 
followed.  First,  the  hardware  and  software  designs  must  be  developed  and  im- 
plemented to  a point  where  reliable  and  stable  outputs  are  provided  without 
any  actual  course  materials  on  the  system.  Secondly,  adequate  provisions  should 
be  made  within  the  contract,  if  an  outside  contractor  is  developing  the  system, 
for  an  appropriate  job  stream  loading  of  software  on  the  system  to  test  relevant 
applications.  It  is  only  following  the  testing  and  debugging  of  the  hardware 
and  software  that  course  development  should  proceed  on  the  system.  Some  general 
off-line  development  of  course  materials  could  be  initiated  prior  to  the  im- 
plementation on-line.  However,  it  is  not  appropriate  to  do  a great  deal  of 
course  development  work  without  knowing  the  formatting  requirements  and  how 
the  material  will  appear  on  the  terminal  device.  If  too  much  is  accomplished 
before  a terminal  is  available,  then  a good  deal  of  revision  will  have  to  be 
performed  and  the  course  development  efforts  could  suffer  as  a result.  As 
sections  of  the  instructional  material  are  developed,  they  can  be  tried  out  and 
validated  with  selected  students.  At  this  time,  too,  the  instructional  model 
can  be  tested  for  consistency  and  adequacy  on  the  system.  Only  after  the  ma- 
terials have  been  formatively  evaluated  and  debugged  should  the  course  and  the 


139 


hardware/sof tware  system  be  exposed  to  actual  operational  trainees.  Some  of  the 
negative  attitudes  on  the  part  of  instructors  and  frustrations  on  the  part  of  the 
students  in  the  CTS  project  could  have  been  eliminated  if  this  system  develop- 
ment sequence  had  been  followed. 

• Employ  an  evaluation  model  that  is  consistent  with  the 
project  purpose(s). 

An  appropriate  evaluation  model  is  required  that  will  focus  the  data 
collection.  Determine  whether  the  type  of  evaluation  is  formative  in  which 
case  its  purpose  is  to  ensure  that  the  system  effectively  teaches.  This  will 
entail  an  iterative  process  of  development,  debug,  testing  and  revision  of  the 
course  materials.  Such  should  be  the  case  in  a prototype  implementation.  On 
the  other  hand,  if  one  is  in  an  operational  mode,  then  a summative  evaluation 
would  be  appropriate.  In  the  case  of  the  initial  CTS  purpose,  the  OTP  included 
a cost-effectiveness  evaluation  based  upon  a summative  evaluation  of  a CAI 
system.  This  was  premature  on  two  counts:  (1)  A formative  evaluation  was 
required  at  that  time,  and  (2)  when  the  purpose  changed  to  CMI,  a different 
evaluation  model  should  have  been  designed.  In  the  CTS  implementation,  the 
purposes  were  not  universally  agreed  upon.  It  is  clear  that  different  expec- 
tations existed  and  as  a result  some  of  the  School  personnel  expected  a summative 
type  of  evaluation,  yielding  cost-effectiveness  conclusions.  The  problem  is 
that  cost-effectiveness  projections  from  a prototype  system  are  subject  to  high 
error.  In  a prototype  project,  management  should  be  aware  of  the  pressures  by 
training  system  sponsors  who  are  hungry  for  evidence  upon  which  to  base  their 
decisions,  and  should  resist  demands  for  a premature  cost-effectiveness  analysis. 
In  the  future  the  Army  would  do  well  to  establish  guidance  for  performing 
adequate  and  timely  evaluations  and  for  using  evaluation  models  appropriate  to 
the  given  project  purpose.  With  respect  to  costing  analyses,  there  should  be 
adoption  of  uniform  costing  categories  based  upon  whether  or  not  the  system  is 
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in  a development  or  an  operational  mode.  A commitment  must  also  be  made  at 
the  project  s outset  for  collecting  such  data  in  a consistent  and  comprehensive 
manner.  Along  with  this  guidance  effectiveness  criteria  should  be  established 
at  the  targeted  implementation  site  upon  which  to  evaluate  the  effects  of 
training.  In  CTS  training  effectiveness  could  not  be  determined  because  system 
stability  and  effectiveness  were  not  separatable  from  the  training  effects  on 
students . 

• Employ  a staffing  mix  consistent  with  project  needs. 

In  the  case  of  CTS,  a qualified  •’-disciplinary  team  required  to  perform 

the  various  hardware /software  functions,  instructional  modeling,  and  course 
development  was  established  early  in  the  project.  Such  an  approach  should  be 
continued  in  the  future  and  the  mix  should  be  appropriate  to  the  type  of  project. 
There  is  no  need  to  have  a large  hardware  staff  if  you  are  going  to  incorporate 
an  already  stabilized  hardware  system.  A part-time  consultant  or  contractor 
might  be  sufficient.  On  the  other  hand  with  respect  to  evaluation  and  costing 
analysis,  there  should  be  dedicated  staffing  for  those  purposes  to  include 
adequate  support  for  data  collection.  If  comparative  analyses  are  to  be  made, 
then  there  should  be  equivalent  support  provided  to  collect  data  from  the 
alternative  training  systems  against  which  the  innovation  is  to  be  compared. 

In  the  case  of  the  CTS  project,  not  enough  resources  were  allocated  for  these 
functions. 

• Permanence  of  key  personnel  is  essential  to  timely  completion  of 
a complex  project. 

One  of  the  problems  the  Services  continually  face  is  that  of  personnel 
turbulence.  It  is  particularly  critical  that  turbulence  be  held  to  a minimum 
when  attempting  to  implement  such  a complex  innovation  as  computer-based  train- 
ing systems.  In  the  case  of  CTS  when  the  change  in  site  location  occurred 
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there  was  a loss  of  key  personnel  familiar  with  the  software  and  hardware  as 
well  as  with  the  project's  management  process.  In  the  future  the  Army  would  do 
well  to  provide  sufficient  extensions  of  duty  tours  to  accommodate  completion 
of  such  prolects. 

Training  Site/Project  Interface:  Managing  the  Implementation 

Key  points  to  consider  related  to  this  area  of  concern  are  the  following: 

• Lit eracy  and  orientation  program  Is  required  prior  to  site  Installation . 

One  of  the  most  important  problems  to  overcome  is  to  insure  that  the  users 
at  the  target  site  understand  the  nature  of  the  computer-based  innovation. 
Without  doubt,  the  administration  of  a School  and  the  required  support  for 
Implementing  high  technology  such  as  computer-based  training  will  require  an 
adjustment  and  accommodation  that  may  be  disruptive  to  the  old  way  of  doing 
things.  The  resistance  to  a large-scale  innovation  like  CTS  can  be  overcome 
by  a tailored  computing  literacy  program  and  orientation  for  all  levels  of 
management.  Clearly,  efforts  are  required  in  the  hardware/sof tware  area  to 
prepare  a site  for  installation  of  the  equipment.  There  is  also  a need  to 
prepare  site  personnel  for  system  implementation  by  having  them  understand  what 
effects  the  computer  will  have  on  their  activities.  User  personnel  can,  in 
turn,  suggest  new  ways  in  which  the  innovation  might  be  extended  to  accomplish 
other  objectives.  There  can  be  a continual  feedback  of  good  ideas  and  coopera- 
tion once  an  initial  understanding  is  accomplished.  This  must  be  done  in  the 
future  if  satisfactory  implementation  of  any  innovative  system  is  to  be 
accomplished . 


142 


• Commitment,  lines  of  control  and  participatory  management  Involving 

the  designated  user  site  must  be  established  prior  to  installation. 

The  arrangements  at  Ft.  Gordon  provided  support  personnel  and  facilities 
to  the  CTS  project.  There  was,  however,  no  contribution  of  staffing  and  dollars 
and  no  degree  of  control  over  the  project  implementation.  This  resulted  in  the 
perception  of  CTS,  as  indicated  from  the  various  interviews  and  survey  forms, 
as  an  outside  imposition  on  the  School.  The  initial  Memorandum  of  Understanding 

(see  Attachment  #10)  indicated  that  there  would  be  no  real  control  or  authority 
by  the  School  until  the  test  and  evaluation  period  was  complete.  In  the  future, 
a new  training  system  implementation  should  be  under  more  direct  control  and 
management  of  an  informed  and  receptive  user  site. 

• A single  chain  of  command  for  project  management  at  the  designated  site 

is  necessary. 

As  noted  earlier  the  CTS  project  had  a dual  chain  of  command  with  School 
personnel  reporting  to  their  chiefs,  CTS  Field  Office  personnel  reporting  to 
their  chiefs,  and  the  development  of  project  products  based  upon  cooperative 
undertaking  from  both  sources.  This  duality  was  responsible  for  much  ambiguity 
and  lack  of  desire  to  take  on  responsibility.  It  was  apparent  in  many  of  the 
interviews  that  no  one  was  willing  to  take  a risk  and  be  held  responsible  un- 
less absolutely  necessary.  Thus,  only  when  a strong  personality  decided  that 
something  had  to  be  accomplished  was  much  direction  given  to  project  develop- 
ment. This  must  be  avoided  in  the  future.  The  Project  Manager's  Office  must 
be  maintained  at  the  implementation  site.  It  is  our  considered  opinion  that 
many  difficulties  could  have  been  avoided  in  the  CTS  implementation  if  this  had 
been  the  case.  Without  such  collocation,  ambiguities  in  communication  will 
result. 
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Integrate  the  chain  of  command  (management)  within  the  trainln; 
directorate  of  the  use r site . 

When  Implementing  a computer-based  training  system  at  a School,  a single 
person  should  be  in  charge  who  is  located  within  the  Office  of  the  Deputy 
Commandant  of  the  School.  Teams  are  appropriate  but  the  management  has  to  be 
vested  in  one  individual  high  enough  up  in  the  command  structure  of  the  School 
so  that  his  responsibility  is  matched  by  the  authority  of  the  Deputy  Commandant. 
Within  the  TSD  context,  representatives  of  each  major  activity  should  be 
involved.  However,  all  should  be  under  the  overall  direction  and  authority  of 
a single  individual.  Support  personnel  are  required  from  the  computer  science 
and  engineering  groups  but  they  are  indeed  subordinate  to  the  purpose  of 
implementing  a training  system  and  should  not  be  in  a control  position  with 
respect  to  that  Implementation. 

• Frequent  meetings  are  necessary  to  monitor  the  atmosphere  of 
expectation  and  understanding. 

One  must  ensure  that  administrative  and  support  personnel  not  only  accept 
the  fact  of  change  but  that  they  are  allowed  to  express  openly  and  frequently 
the  problems  that  plague  their  work.  Coordination  of  team  efforts,  the 
adherence  to  priorities,  meeting  various  contingent  deadlines,  etc.,  require 
such  meetings  to  be  scheduled  regularly  and  that  continuous  communication 
exist  among  the  project  management,  contractors,  sponsors,  and  users. 
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ATTACHMENT  //I 


DEPARTMENT  OF  THE  ARMY 

MADQUAHTtma  UNITED  STATES  CONTINENTAL  ARMY  COMMAND 
FORT  MONROE.  VIRGINIA  *3381 


ATIT-STM  29  June  1972 

SUBJECT:  Computer  Assisted  Instruction  Prototype  Program  Implementation 


Commanding  General 

US  Army  Signal  Center  & School 

Fort  Monmouth,  New  Jersey  07703 


1.  Reference  letter,  A TSCC-DOI-CAI,  HQ  USASCS,  7 June  1972,  subject: 
CONARC  CAI  Task  Group  Report,  April  1972. 

2.  Based  on  the  additional  information  contained  in  referenced  letter, 

USASCS  has  been  selected  as  the  site  for  the  CAI  Prototype  System  Implemen- 
tation. 

3.  The  CONARC  project  manager  (PM)  for  the  CAI  Prototype  System  will  be 
designated  from  within  the  staff  of  the  USASCS.  The  PM  will  manage  the 
program  for  CONARC  and  also  be  executive  secretary  of  the  DA  established 
CAI  Steering  Group.  A list  of  members  of  the  CAI  Steering  Group  is  at 
Inclosure  1 (Annex  C-fe). 

4.  Computer  Systems  Support  and  Evaluation  Command  will  assist  in  the 
development  of  hardware  specifications  and  will  perform  required  competitive 
procurement  actions  in  coordination  with  the  PM. 

5.  Office  of  the  Chief  of  Research  and  Development  will  provide  applied 
research  and  development  support  for  the  CAI  Prototype  System  Program. 

The  extent  and  type  of  support  to  be  provided  are  under  negotiation. 

6.  USASCS  is  requested  to  identify  the  resources  necessary  to  perform 
this  mission: 

a.  Funds  required  will  oe  identiiied  oy  program  elements  and  by 
fiscal  year. 

b.  Justification  for  personnel  essential  to  the  operations  of  the 
CAI  Prototype  System  will  be  suomitted  to  this  headquarters  in  accord- 
ance with  CON  Reg  1-45. 


ATIT-STM 

SUBJECT:  Computer  Assisted  Instruction  Prototype  Program  Implementation 


7.  This  headquarters  will  be  kept  informed  of  the  progress  of  the  project 
through  reports  generated  by  the  CAI  Steering  Group,  and  other  reports  as 
' required. 

f FOR  THE  COMMANDER: 


(SIGNED) 

1 In  cl  IRA  A.  HUNT,  JR. 

as  Major  Gene ral,  GS 

Deputy  Chief  of  Staff 

CF:  for  Individual  Training 

HQDA  (DACS-CMS) 
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ATTACHMENT  #2 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 

The  purpose  of  this  survey  is  to  gather  Information  concerning  the  develop- 
ment of  Computerized  Training  System  instructional  materials  for  the  31E20, 
31J20,  and  35L20  courses.  Please  answer  all  items  in  this  survey  as  factu- 
ally and  completely  as  possible,  and  with  complete  candor.  Your  responses 
will  be  held  In  strictest  confidence.  If  you  feel  the*  you  are  not  In  a 
position  to  answer  a particular  item  because  you  have  not  been  closely 
associated  with  the  CTS  project,  please  circle  the  Item  number  and  leave 
It  blank.  Your  comments  or  suggestions  will  be  greatly  appreciated. 

Date 


PI  ease 

indicate  (^)  your  position 

re  1 at  I ve  to 

CTS: 

a . 

1 nstructor 

e. 

Division 

Chief 

b. 

Instructional  Programmer_ 

f . 

Education 

i Special  1st 

c. 

Section  Chief 

Training 

Special  1st 

d. 

. 

Course  Chief 

h. 

Other 

PLEASE  RECORD  YOUR  ANSWER  OR  CHECK  (^)  THE  ALTERNATIVE  WHICH  BEST  EXPRESSES 
YOUR  REACTION  TO  EACH  ITEM  THAT  FOLLOWS. 
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COURSE  MATERIALS  DEVELOPMENT  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 


SECTION  I.  Fitting  Existing  Self- 
Paced  Training  Materials  into  the 
CTS  Model . 

1.  The  Instructional  model  Is  flex- 
ible enough  to  accomodate: 


a.  Course  strategies 


Subject  matter. 


The  CTS  instructional  model  Is 
compatible  with  the  systems 
engineering  of  training  regulre- 
ments  of  course  development. 


No  revision  of  systems  engineer- 
ing of  training  documents  was 
necessary  to  fit  previously  de- 
veloped self-paced  materials  into 
the  CTS  instructional  model. 


Introduction  of  on-line  testing 
(pretests/posttests)  reguirod 
major  revamping  of  learning  units 
( lesson  plans) . 


The  CTS  model  regu i red  little 
change  in  Instructional  materials 
because  of  the  limited  amount  of 
on-l i ne  I nstruct Ion . 
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CD 
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Comments 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

Strongly  Agree 

— 

Agree 

Neutra I 

| 

Q) 

0) 

L. 

CD 

(TJ 

If) 

5 

0) 

(D  , 

L. 

CD 
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CD 

c 

o 

L. 

4- 

tr> 

Comments 

6.  The  CTS  Instructional  model  varied 
only  slightly  from  the  on-going 
instructional  model. 

| 

7.  The  CTS  model  was  designed  pri- 
marily for  instruction  in  the  CAI 
(student  interacts  directly  with 
terminal)  mode  and  does  not  lend 
Itself  well  to  a course  that  is 
heavily  CMI  (material  presented 
off-line  under  computer  manage- 
ment ) . 

8.  Using  the  CTS  instructional  model 
has  accelerated  the  conversion  of 
training  materials  to  the  CTS 
mode . 

1 

1 

9.  The  crs  Instructional  modol  is  espe- 
cially adaptive  for  the  use  of 
multimedia  lesson  presentations. 

1 

1 

1 

10.  The  CTS  instructional  model  has 
sufficient  latitude  to  accommo- 
date the  various  learning  alter- 
natives utilized  in  the  course. 

| 

1 

11.  The  CTS  instructional  model  is 
designed  to  accommodate  the 
dynamic  training  materials 
(where  the  learning  path  is  de- 
termined by  student  performance 
rather  than  predetermi ned  apti- 
tudes) . 

1 

1 
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COURSE  MATERIALS  DEVELOPMENT  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  to  I low  in 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 
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Comments 


12.  The  CTS  Instructional  model  en- 
ables the  system  to  make  deci- 
sions concerning  student  progres- 
sion that  were  formerly  made  by 
the  instructor. 


SECTION  II.  Problems  Encountered 
when  Entering  Training  Materials  on 
Line. 

13.  1 nstruct iona I programmers  must 

oversee  the  input  of  course 
materials  by  Instructional  Pro- 
gram Entry  Specialist  (IPES) 
and/or  Clerk-Typists. 


14.  Delays  were  experienced  in 
entering  instructional  mate- 
rials on-line  because  of  the 
shortage  of  clerk-typists  and 
IPES  personnel . 


15.  Clerk-typist/IPES  personnel 

experienced  little  or  no  diffi- 
culty in  entering  lesson  mate- 
rial or  logic  coding  into  the 
system. 


16.  The  amount  of  CAl  in  the  course 
has  been  limited  by  the  program- 
med (4-1)  student  to  terminal 
ratio  in  the  classroom. 
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COURSE  MATERIALS  DEVELOPMENT  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

Strongly  Agree 

Agree 

Neutra I 

Di  sagree 

— 

Strongly  Disagree 

Comments 

17.  Compiling  the  lesson  material 

after  being  entered  on-line  has 
caused  considerable  delay  in 
loading  the  courseware  into  the 
system. 

1 

18.  To  effectively  prepare  a unit  of 
instruction,  the  same  Instruc- 
tional Programmer  should  be  the 
author,  complete  the  logic  coding, 
enter  (or  supervise  the  entry)  of 
the  material  into  the  system  and 
accomplish  debugging. 

19.  Logic  coding  of  lesson  materials 
has  caused  relatively  few  prob- 
lems when  entering  materials 
into  the  system. 

i 

1 

20.  Editing  materials  on-line  has 
been  compounded  by  IPES/clerk- 
typist  typing  errors. 

1 

1 

1 

1 

21.  Little  difficulty  was  experienced 
when  deleting  or  adding  new  or 
revised  CTS  instructional  mate- 
ria 1 s on-l i ne. 

1 

1 

1 

22.  A complete  CTS  unit  of  instruc- 
tion can  be  changed  overnight 
eliminating  any  delay  in  student 
progress . 

I 
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23.  The  average  waiting  time  (queue  time  and  compile  time)  for  the  system 
to  compile  an  instructional  Class  I source  code  has  been 

a.  10  minutes 

b.  20  minutes 

c.  30  minutes 

d.  40  minutes 

e.  50  minutes 

f.  Other  (show  time) 

24.  What  problems  were  encountered  when  inputting  and  "saving"  lesson 
material  during  the  process  of  entering  courseware  on  line? 


25.  How  much  adverse  effect,  if  any,  has  computer  down  time  had  on  entering 
of  materials  on  line?  Explain. 


26.  In  your  opinion,  what  are  the  advantages  and  d i sadvantages  In  using 
the  CTS  Instructional  model  when  converting  existing  training  materials  to 
CTS? 


Advantages 

Di  sadvantages 

COURSE  MATERIALS  DEVELOPMENT  SURVEY 
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27.  What  is  your  frank  opinion  concerning  the  relative  worth  of  the  CTS 
Instructional  model? 


28.  Did  waiting  for  system  restarts  delay  the  preparation  of  instructional 
materials?  Explain. 


29.  What  special  problems  did  you  encounter  when  logic  coding  lesson  mate- 
rials? 


30.  Was  there  any  limitation  to  the  i :mber  of  IPES's  that  coula  enter  mate- 
rials into  the  system  simultaneously? 

Yes  No 

If  yes,  what  was  the  limitation? 


31.  If  the  answer  to  item  30  above  was  yes,  how  did  this  hamper  the  loading 
of  courseware  into  the  system? 
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Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 


M\ 


Comments 


SECTION  III.  Test  Analysis. 

32.  On-line  pretests  are  effective 
in  diagnosing  student  knowledge 
of  the  instructional  unit. 


The  results  of  the  pretests  are 
good  predictors  of  student  ac- 
comp I i shment . 


On-line  pretests  are  not  a 
factor  in  routing  students 
through  the  course. 


On-line  pretest  results  have 
little  impact  on  student 
performance. 


36.  Since  pretests  are  optional 
most  students  elect  not  to 
take  the  pretest. 


Of  the  students  who  elect  to 
take  the  pretests,  the  majority 
fail. 


Posttests  have  been  effective 
in  measuring  student  accomplish- 
ment . 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
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Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

Strongly  Agree 

Agree 

Neutra 1 

Q) 

V 

B) 

<D 

U) 

Q 

<D 

<D 

cn 

ro 

in 

Q 

>~ 

O) 

c 

O 

L. 

+- 

CO 

Comments 

39.  Distractor  counts  have  identified 
weaknesses  in  the  instructional 
material  per  se. 

40.  Pretest  and  posttest  distractor 
counts  have  pinpointed  deficien- 
cies inherent  in  the  test  questions. 

1 

1 

41.  Posttests  have  proved  to  be  highly 
successful  in  routing  the  student 
into  his  proper  learning  alterna- 
tive. 

42.  Distractor  counts  have  enabled 

instructional  programmers  to  make 
timely  revisions  to  questions 
and  training  materials. 

43.  Because  of  misspelling  and  im- 
proper phrasing,  more  unantici- 
pated responses  have  surfaced 
than  expected. 

: 

44.  Unanticipated  responses  are 
useful  when  revising  instruc- 
tional material. 

45.  How  can  test  distractor  counts  and  collection  of  unanticipated  re 
sponses  be  used  to  revise  training  materials? 
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SECTION  IV.  Conversion  of  Training  Materials  to  CTS. 

This  section  will  focus  on  the  conversion  of  existing  instructional  materials 
to  the  CTS  instruct iona I mode.  The  task  will  be  the  basic  instructional  ele- 
ment for  analysis.  Two  areas  will  be  addressed:  (1)  tasks  with  the  greatest 
number  of  hours  of  CA I , and  (2)  tasks  devoted  primarily  to  CMI. 

46.  Please  show  the  break  out*of  CAI  (on-line  instruction  and  testing)  and 
CMI  (off-line  instruction  under  computer  management)  within  your  course. 

SPECIAL  NOTE:  One  hour  of  CAI  or  CMI  is  construed  to  be  that  amount  of  in- 
structional material  programmed  to  be  completed  by  the  student,  with  an  average 
Progression  Index  (P.l.)  of  1.00,  in  one  academic  (50  minutes)  hour. 

Course:  (circle)  31E20  31J20  35L20 

Academic  Hours  in  Course:  (Do  not  include  COBET)  

Task  ft Hours  in  Task Hours  in  CAI Hours  in  CMI 

1  

2  

3 

4 ______  

5 

6 

7 

8 

9 

10  

11  

12  

13  

14  


•Refer  to  Course  Profile 


162 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Task  § 


Hours  in  Task 


Hours  in  CAI 


Hours  in  CM  I 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
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SECTION  IV.  Conversion  of  Training  Materials  to  CTS. 

48.  Based  on  CTS  time  logs,  request  the  time  required  for  conversion  of 
training  materials  of  four  tasks  (familiar  to  you)  to  CTS  be  calculated  as 
indicated  below.  Select  a medium  length  and  long  task  containing  the 
greatest  number  of  CAI  hours,  and  a medium  and  long  task  devoted  primarily 
to  CM  I for  time  analysis. 

a.  Lona  Task  - CAI 


Task  Hrs  in  Hrs 
No.  I Task  CAI 


Med i urn  Task 


Task  Hrs  in  Hrs 
No.  Task  CAI 


CTS  Instruct iona 

Materia  1 s 

1 . 

Unit  Development 
( 1 nc 1 ude  a 1 1 on- 

i ne  materia  1 ) 

2. 

Logic  Coding 

3. 

Editing  and  Debugging 

B 

Review  and  Rev  is 

on 

5. 

Other 

CTS  Instruct iona 

Materia  1 s 

1 . 

Unit  Development 
( 1 ncl ude  a 1 1 on- 

i ne  mater i a 1 ) 

2. 

Logic  Coding 

3.  Editing  and  Debugging 


. Review  and  Revision 


5.  Other 
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COMMENTS : 
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SECTION  V.  Time  Required  to  Prepare  One  Hour  of  CAI  (on-line  instruction 
and  testing). 

SPECIAL  NOTE:  One  hour  of  CAI  is  construed  to  be  that  amount  of  instruc- 
tional material  programmed  to  be  completed  by  the  student,  with  an  average 
P. I . of  1.00,  in  one  academic  (50  minute)  hour. 

49.  Based  on  CTS  time  logs,  in  a task  that  includes  9 or  more  hours  of  CAI, 
what  was  the  average  conversion  (the  adaptation  of  on-going  self-paced  in- 
structional material  to  the  CTS  mode)  time  per  hour  of  CAI?  Explain. 


50.  Based  on  CTS  time  logs,  what  was  the  average  conversion  (the  adaptation 
of  on-going  self-paced  instructional  material  to  the  CTS  mode)  time  for  one 
hour  of  CAI,  exclusive  of  testing.  Explain. 


51.  What  is  the  average  conversion  (the  adaptation  of  on-going  self-paced 
instructional  material  to  the  CTS  mode)  time  required  to  enter  one  hour  of  CAI 
(include  testing)  on-line,  debug  and  validate  the  material? 
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52.  What  is  the  difference  in  the 
self-paced  instructional  materials 
debug,  and  validate  an  hour  of  CAI 
that  does  not?  Explain. 


conversion  (the  adaptation  of  on-going 
to  the  CTS  mode)  time  required  to  enter, 
instruction  that  includes  testing  and  one 


53.  Do  the  time  est 
(initial  development 
student)  required  to 
factors  only? 


mates  in  paragraphs  49  and  50 
of  training  material  not  prev 
prepare  an  instructional  unit 


include  the  basic  research 
iously  presented  to  the 
, or  are  they  conversion 


54.  If  just  convers 
accomplish  the  basic 
previously  presented 


ion  factors,  how  much  additional  time  must  be  allotted  to 
research  (initial  development  of  training  material  not 
to  the  student)  required  for  one  hour  of  CAI? 
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55.  Based  on  your  answers  to  questions  49  through  54,  what  is  your  best 
estimate  of  the  total  time  required  to  prepare  one  complete  hour  of  inter- 
active CAI,  exclusive  of  any  testing?  Explain. 


56.  Do  you  feel  the  time  estimate  cited  in  question  55  would  apply  to  all 
segments  of  instructional  material?  Explain. 


57.  Do  you  be  I ieve 
reasonable  accuracy 


the  time  estimate 
to  predict  future 


given  in  question  55 
CAI  development  requ 


can  be  used  with 
rements? 


COURSE  MATERIALS  DEVELOPMENT  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


SECTION  VI.  Time  Required  to  Develop  One  Hour  of  CAI. 

SPECIAL  NOTE:  One  hour  of  CAI  is  construed  to  be  that  amount  of  instruc- 
tional material  programmed  to  be  completed  by  the  student,  with  an  average 
P. I . of  1.00,  in  one  academic  (50  minute)  hour. 

58.  Based  on  your  time  estimate,  in  answer  to  question  49,  show  the  breakout 
of  time  required  to  convert  one  hour  of  self-paced  training  material  to  CAI. 


Breakdown  of  Time  to  Complete  One  Hour  of  CAI 

1 tern 

Your  Estimate 
Hours 

a.  Planning,  strategy,  lesson  unit  outline 

b.  Authorizing  text  (writing,  typing,  coding) 

c.  Training  aids,  photography,  graphics 

d.  Debugging,  revisions,  evaluation,  coordination 

e.  Small  and  large  group  validation 

f.  Reference  literature,  advanced  worksheets  and 
special  texts 

Total  Estimated  Hours 

ATTACHMENT  ft 3 


I 

I 

I 

I 
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13  December  1976 

The  purpose  of  this  survey  is  to  gather  information  concerning  revision 
and  validation  of  CTS  course  materials,  as  well  as  alternate  training 
material  requirements,  related  to  the  Computerized  Training  System  (CTS) 
in  the  31E20,  31J20  and  35L20  courses. 

Please  answer  all  items  in  this  survey  with  complete  candor.  Your  re- 
sponses will  be  held  in  strictest  confidence.  The  results  of  this  survey 
will  enable  the  revision  and  validation  of  training  materials  to  be  more 
effective. 

If  you  feel  that  you  are  not  in  a position  to  answer  a particular  question 
because  you  have  not  been  closely  associated  with  the  CTS  project,  please 
circle  the  item  number  and  leave  it  blank.  Your  comments  or  suggestions 
will  be  greatly  appreciated. 

Place  a check  (v'T  mark  by  your  location  and  position  in  your  organiza- 
tion. You  do  not  have  to  sign  this  form. 

Locat i on 


Training  Development  Directorate  (Staff) 
Department 

Training  Design  Division 


Education  Specialist  

Training  Specialists  

Instructor  

Supv  Training  Instructor 


l 


PLEASE  RECORD  YOUR  ANSWER  OR  CHECK  ( v/5  THE  ALTERNATIVE  WHICH  BEST 
EXPRESSES  YOUR  REACTION  TO  EACH  ITEM  THAT  FOLLOWS. 


! 


REVISED  AND  ALTERNATE  TRAINING  MATERIALS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

1 

Agree 

Neutra 1 

Di sagree 

Strongly  Disagree 

Comment 

SECTION  1.  Revision  and  Validation  of 
Training  Materials. 

1.  CTS  provides  more  flexibility  in 
revising  instructional  materials 
than  does  the  self-paced  system. 

2.  Little  difficult  has  been  ex- 
perienced in  revising: 

a.  Individual  displays. 

1 

1 

1 

1 

1 

■ 

b.  Tests. 

■ 

■ 

■ 

■ 

■ 

B 

c.  Units  of  instruction. 

■ 

■ 

■ 

■ 

■ 

B 

d.  Flow  of  instruction 

■ 

■ 

■ 

■ 

3.  The  CTS  review  process  has  re- 
duced the  time  normally  required 
for  introducing  new  or  revised 
materials  into  the  classroom. 

1 

1 

1 

1 

1 

4.  CTS  instructional  materials  can 
be  introduced  into  the  course 
without  the  usual  printinq  re- 
quirement  procedures. 

Ai 
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Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

strongly  Agree 

! 

0 

0 

i_ 

rj) 



(T3 

i- 

+- 

Z3 

0 

JZ 

1 

0 

0 

L_ 

cn 

(V 

cn 



1 

Comments 

5.  It  is  easier  to  validate  instruc- 
tional materials  on-line  than  it 
is  to  validate  conventional  or 
other  self-paced  printed  mate- 
rials. 

6.  It  takes  less  time  to  review  CTS 
instructional  materials  on-line 
than  it  does  to  review  self-paced 
training  materials. 

1 

1 

■ 

7.  Minor  revisions  to  CTS  instruc- 
tional materials  are  accomplished 
during  the  on-line  review  process. 

1 

1 

1 

i 

8.  Using  students  to  validate  CTS  on- 
line training  materials  has  been 
helpful  in  revising  instructional 
units. 

1 

1 

1 

1 

9.  The  use  of  small  and  large  student 
groups  to  validate  revised  CTS  in- 
structional units  has  not  impaired 
the  review  process. 

1 

1 

1 

1 
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Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 


SECTION  II.  Back-up  Training  Materials. 

10.  Was  it  necessary  to  prepare  back-up 
training  materials  to  be  used  ex- 
clusively to  cover  computer  down 
time?  If  yes,  please  explain. 

Yes 

No 

■ 1 1 rn  rrnT— i 

11.  Were  course  personnel,  other  than 
instructional  programmers,  used 
to  prepare  back-up  training  mate- 
rials to  cover  computer  down  time? 

If  yes,  please  explain. 

12.  Did  the  requirement  to  develop 
back-up  training  materials  in- 
crease the  workload  for  the  course 
writers?  If  yes,  what  percentage? 

13.  Could  CTS  instructional  materials, 
prepared  for  entry  on- 1 ine,  be 
used  as  back-up  material  for  in- 
instruction during  down  time? 

14.  Was  it  possible  to  use  computer 
printouts  of  on-line  instruc- 
tional material  as  back-up 
training  material  during  computer 
down  time?  Please  explain.' 
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I 

I 


15.  In  determining  course  resource 
requirements,  should  additional 
personnel  be  programmed  to  handle 
the  preparation  of  back-up  mate- 
rials? Please  explain. 

Yes 

No 

| 

Exp  1 anat ion 

16.  Can  CTS  instructional  materials 
plus  back-up,  be  prepared  with 
the  same  number  of  personnel  re- 
quired to  prepare  instructional 
materials  for  the  conventional 
self-paced  mode. 

17.  Do  you  think  there  is  a legiti- 
mate requirement  to  develop 
back-up  instructional  materials 
for  all  CTS  on-line  instruction? 

If  yes,  please  explain. 

18.  Do  your  students  have  a copy  of 
the  training  objectives  when 
they  start  training?  If  not,  why: 

19.  Do  you  pretest  your  revised 
training  packages? 

1 

- 

20.  Do  you  val idate  your  units  of 
instruction  individually? 

21.  Do  you  validate  the  annex  or  task 
in  its  entirety? 
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22.  Are  statistical  procedures 

used  in  validating  your  train- 
ing packages?  If  not,  explain 
how  you  va 1 i date. 

Yes 

m 

i 

Exp  1 anation 

23.  Do  you  think  there  is  a legiti- 
mate requirement  to  develop 
back-up  instructional  materials 
for  all  CTS  on-line  instruction? 

If  yes,  please  explain. 

SECTION  III.  General  Summary. 

24.  After  a task  or  annex  has  been  revised  by  the  instructional  programmer 
how  much  time  is  normally  required  for  review  and  approval  before  these 
materials  can  be  used  in  the  classroom? 

a.  15  days  

b.  30  days  

c.  45  days  

d.  60  days  

e.  Other 


25.  How  much  time  has  been  saved  as  a result  of  implementation  of  CTS 
in  the  modification,  revision  and  validation  of  course  materials? 


a. 

0 

days 

b. 

5 

days 

c. 

10 

days 

d. 

20 

days 

e. 

30+ 

days 
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26.  Have  you  encountered  any  special  problems  in  preparing  CTS  instructional 
materials  not  noted  when  preparing  prior  self-paced  materials? 


27.  How  would  you  change  the  present  system  of  revising  le  son  materials 
to  insure  adequate  back-up  for  CTS  on-line  instruction? 


I 

I 

» 

I 

I 

I 

I 


28.  What  problems  peculiar  to  CTS  were  encountered  when  using  small  and 
large  student  groups  for  validation  of  training  materials? 
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COURSE  DEVELOPMENT  SUMMARY  SHEET 

" * 

NAME:  POSITION:  DATE: 


Course (s)  worked  on: 

Course  Annex(es)/Task(s)  Position  Time  Period 

1.  

2.  , ■ 

3.  

4.  

5.  ■ • 

6. 


* NAME: 


COURSE:  1 TIME  PERIOD: 

Estimated  days  worked  in  course:  

Annex/task:  

# TAISs  in  Annev:  

# TAISs  worked  on:  __ 

Estimated  H days  worked  on  annex:  j 

Off-line  Presentation  On-line  Presentation 

a.  Pre-  and  Post-Tests  1 

b.  EPIS  Tests  

c.  Performance  Tests  r 

d.  Practice  Problems/Materials  » 

e.  Performance  Guides  

f.  Lesson  Materials  such  as  special 

texts,  or  on-line  presentations  * 

g.  Remediation  Activities  

h.  A/V  Materials  ' 

i.  Training  Aids  ' ? 

j.  Directions  I 

k.  Lesson  Outlines  

l . Other  ' | 

% of  time:  % of  time:  | 

t 

For Course,  % thrown  out  due  to  POI  changes.  | 

I | 
I 1 
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COURSE:  ' TIME  PERIOD: 

Annex/Task:  


Off-line  Presentation  On-line  Presentation 


a.  Planning,  Course  Outlines, 
Strategies 

b.  Original  Authoring  (writing, 

• typing,  coding) 

c.  Converting  Existing  Materials 

d.  Modifying  Materials  due  to 
POI  Changes 

e.  Reviewing,  Debugging,  Testing, 
etc..  Materials 

f.  Revising  Materials 

g.  Coordination 

h.  Other 


NAME 


DATE  OF  INTERVIEW 


What  feedback  is  available  for  you  to  use  in  revising  instructional  materials 
and  tests  (e.g.,  Aug*  PI  by  Task  or  TAIS)? 


NAME 


COURSE 


Tel.  No. 


8.6.1  What  special  qualifications  are  required  by  the  instructional  support 
staff? 

Instructional  Programmers 


b.  Course  Development  Personnel 


c.  Computer  Service  Personnel 


d.  1PES 


e.  Other  (specify) 


NAME 


COURSE 


Tel . No. 


8.C.2  Wh at  administrative  and  personnel  costs  were,  incurred  to  establish  in- 
service  training  programs  (for  CTS  Instructors  and  Workshops  for 
Instructional  Programmers  and  IPES  personnel)? 

Recommended  Data  Source : Input  from  ISA5IG  Data  Gy  stems  Division 
Comments:  Additional  interview  and/or  survey  questions  appear  necessary. 


COURSE 


Tel.  No. 


7.2.1  What  difficulties  have  been  encountered  in  fitting  the  previously  developed 
self-paced  instructional  materials  into  the  CTO  instructional  model? 


IH7 


NAMK 


COURSE 


Tel . No . 


7.2.3  What  special  problems , if  any,  were  encountered  when  entering  ( inputting ) 
training  materials  on  line? 
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NAME  COURSE To]  . No. 

9.1  What  is  the  average  training  time  hy  task  and  annex  for  each  CTO  course 
What  is  the  standard  deviation ? How  many  graduates  were  there  in  each 
course? 


I 

NAME  _ COURSE  To].  No. _ ^ 

9.2  What  wan  the  number  and  percentage  of  students  who  failed  to  graduate 
from  each  CTS  course? 


i 
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OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 

DATE 

The  purpose  of  this  survey  Is  to  collect  Information  on  maintenance  of 
student  records  In  the  courses  utilizing  computers.  This  Information 
will  be  analyzed  with  the  Idea  of  making  record  keeping  simpler  and  easier 
to  ma I nta I n. 

Please  Indicate  your  position  below  by  a check  mark  ( •'T’.  If  more  than  one 
position  applies,  make  additional  <■  ock  marks. 

Dept  Operations  Chief  Course  Chief 

Dept  Education  Specialist  Course  NCOIC 

Dept  Course  Materials  Analyst  Course  Training  Specialist 

Dept  Records  Clerk  Course  Section  Chief 

Division  Chief  Course  Instructor 

Division  NCOIC 

The  following  Is  a glossary  of  terms  which  will  assist  you  In  answering 
the  questions: 

. On-line  Testing  - Using  the  computer  to  administer,  score,  and  record 
a student's  test. 

. Diagnostic  Tests  (quizzes)  - An  Informal  test. 

. Regularly  - At  regular  times  or  intervals. 

. Occasionally  - Now  and  then. 

. Seldom  - On  only  a few  occasions. 

. On-line  - Interaction  of  operating  a terminal  with  the  computer. 

. Downtime  - An  Interval  of  time  when  the  computer  Is  not  productive. 


OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
Indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

Strongly  Agree 

Agree 

Neutra 1 

Disagree 

Strongly  Disagree 

Comments 

SECTION  1.  Weekly  Student  Activity 
Reports. 

1.  CTS  has  reduced  the  time  the 
classroom  Instructor  spends  on 
student  records. 

1 

2.  Weekly  student  activity  printouts 
enable  rapid  analysis  of  student 
accomplishment  in  relation  to 
his  peers. 

3.  The  ability  to  display  student 
records  on  the  terminal 

a.  provides  effective  real  time 
feedback  on  student  progress. 

b.  saves  time  In  assessment  of 
student  needs. 

4.  CTS  has  had  no  impact  on  student 
record  keeping  at  the  division/ 
department  level. 

OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


- 


’ 

n Please  check  the  appropriate  block  to 

indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 


5.  CTS  has  been  successful  in  routing 
students  into  the  learning  alter- 
native dictated  by  their  prior 
accomplishments  in  the  course. 


6.  CTS  reports  enable  course  mana- 
gers to 

a.  assess  trends  as  they  develop 


b.  update  the  Instructional  pro- 
cess with  minimum  delay. 


7.  CTS  student  record  printouts 
enable  the  Instructor  to 

a.  analyze  student  progress 


b.  provide  individual  assistance 


c.  prescribe  remedial  training. 


8.  Recording  no-goes  In  the  weekly 
report  has  assisted  In  managing 
the  student  through  the  course. 


Strongly  Agree 
Agree 
Neutra I 
Disagree 

Strongly  Disagree 


OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


9.  The  objective  of  the  Weekly  Student  Activity  Report  Is  to  provide  the 
course  manager  with  timely  Information  concerning  student  progress  and 
achievement. 

a.  How  well  do  you  think  the  CTS  Weekly  Student  Activity  Report  has 
met  this  objective? 


b.  Has  the  expanded  CTS  Weekly  Student  Activity  Report  provided  you 
with  a better  management  tool?  Explain. 


c.  Is  the  revised  CTS  Student  Graduation  Report  received  In  time  to 
assist  the  course/d  I vis  lon/department  In  completing  the  student 
records?  Explain. 


d.  Has  CTS  improved  the  timeliness  of  the  Student  Graduation  Predic- 
tion? Explain. 


e.  Should  any  additional  Items  be  included  or  deleted  from  the  Weekly 
Student  Activity  Report?  If  yes,  explain. 
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OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


ADDITIONAL 


OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


13.  Does  the  CTS  system  provide  your  course(s)  with  the  necessary  opera- 
tional reports?  If  not,  please  list  and  describe  any  additional  re- 
ports you  think  are  necessary. 


14.  What  CTS  reports  are  not  necessary?  List  and  explain. 


SECTION  III.  Student  Record. 

The  student  record  file  was  created  to  provide  real  time  access  to  Indivi- 
dual student  records,  both  on-iine  and  printouts,  as  required  by  the  pri- 
mary Instructor  or  course  managers. 

1.  Does  the  student  record  contain  the  necessary  Information  to  enable  you 
to 


a.  analyze  a student's  progress?  Explain. 


b.  prescribe  remedial  training?  Explain. 


c.  take  action  to  separate  the  student  from  the  course?  Explain. 


2. 


Have  you  experienced  any  problems  In  using  the  terminal  to  call  up  a 
student's  record?  If  yes,  explain. 


200 


OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


I ^ 

13.  When  you  call  for  a student's  record  on-line.  Is  there  any  appreciable 
time  lag  In  obtaining  the  display?  If  yes,  explain. 

I 

^ 4.  Has  the  printout  of  the  student's  record  been  available  on  a timely 

basis?  If  no,  explain. 


I 

• 5.  Does  the  student  record  contain  the  necessary  data  to  support  the 

counselling  and  guidance  program  In  your  course?  If  no,  explain. 


6.  Do  you  use  the  student  record  printout  to  support  faculty  board  actions? 
Please  explain. 


7.  What  additional  Information  do  you  need  In  the  student  record? 


I 

I 
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OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


SECTION  IV.  Computer  Downtime  and  and  Updating  Student  Activity  Reports. 
(Answer  the  following  questions  with  a "yes"  or  "no"  and  explain  your  answer. 

1.  Does  computer  downtime  Interrupt  the  normal  flow  of  CTS  reports? 


2.  Did  the  computer  downtime  result  In  any  of  the  following: 

a.  Irretrievable  loss  of  student  data? 

b.  Delay  In  student  graduation? 

c.  Distortion  of  students'  progression  Index? 

d.  Hand  processing  of  student  data? 

e.  Overtime  (extra)  work  for  administrative  personnel? 

f.  Delay  In  training  until  the  system  was  restarted? 

g.  Explain  any  special  problems  not  listed  above. 

3.  Did  the  temporary  delay  of  recurring  reports  adversely  affect  student 
training? 

4.  Do  you  consider  computer  downtime  a major  drawback  to  CTS? 
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OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 

1 

5.  Fol 
put 

lowing  computer  downtime,  was  It  necessary  to  manua 
student  data  Into  the  system  to  update  any  of  the 

lly  record  and  In- 
following  reports: 

4 

1 

a. 

Student  Weekly  Activity  Report? 

1 

I 

b. 

Graduation  Prediction? 

c. 

Graduation  Report? 

d. 

Course/Task  Report? 

e. 

Course  Absentee  Report? 

f. 

Company  Absentee  Report? 

9- 

Student  Record  (Data)? 

1 

1 

h. 

« 

Other  (List)? 

1 

1 r 

1 

■ 
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OPERATIONAL  REPORTS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


6.  Reports  listed  below  are  automatically  generated  within  the  computer 
system.  Have  you  experienced  any  problems  with  these  reports  because 
of  loss  of  data  during  downtime? 

a.  TAIS  Report. 

b.  Constructed  Responses. 

c.  Test  Analysis. 

d.  Question  Analysis. 


7.  Were  any  particular  problems,  not  discussed  above,  encountered  In  up 
dating  student  activity  reports  following  computer  downtime? 


8. 


Was  It  necessary  to  maintain  a dual 
records  to  insure  continuity  during 


set  of  manua I I y maintained  student 
downtime? 


9.  Please  list  and  explain  any  problems  encountered  with  the  CTS  Opera 
tlonal  Reports  not  previously  covered. 


\ 
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STUDENT  CLASS  ROSTER 


1.  Are  you  familiar  with  this  report? Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

. ~1s (*• 

a.  Reaularly  available 

b.  o/ewsional ly  available 

' ' c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regul  arly 

b.  Occasional  1 y 

c.  ^ 4Pisel  dom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  Is  your  opinion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d . No  opinion 

6.  The  Information  contained  in  this  report  Is 

a . Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  In  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  Intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 


COMMENTS:  

v • I 


205 


STUDENT  EVALUATION  ROSTER 


1.  Are  you  familiar  with  this  report?  Yes 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

b.  Occasionally  available 

' ’ c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regul  arl  y 

b.  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  _____  Cannot  judge  accuracy  of  report 

5.  What  is  your  op.nion  about  the  format  of  this 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d .  No  opinion 


6.  The  information  contained  in  this  report  is 

a . Essential 

b . Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 


No 


report? 


COMMENTS: 


WEEKLY  STUDENT  ACTIVITY  REPORT 


1.  Are  you  familiar  with  this  report?  Yes 

(If  No,  turn  page  to  next  report.)  * 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

b.  Occasionally  available 

' • c.  Seldom  or  never  available' 

3.  How  frequently  do  you  use  this  report? 

a .  Regularly 

b.  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  _ Minor  inaccuracies 

c.  . Numerous  errors 

d.  ' Cannot  judge  accuracy  of  report 

5.  What  is  your  opinion  about  the  format  of  this 

a . Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d . No  opinion 

6.  The  information  contained  in  this  report  is 

a . Essentia  1 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 


No 


report? 


COMMENTS: 


GRADUATION  PREDICTION 


1.  Are  you  familiar  with  this  report? Yes 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

• b.  Occasionally  available 

• ■ c.  _ Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Kegul  a rl  y 

b.  Occasionally 

c.  Seldom  or  never 


4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  opinion  about  the  format  of  this 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d .  No  opinion 

6.  The  information  contained  in  this  report  is 

a . Essential 

b . Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 


No 


report? 


COMMENTS: 


GRADUATION  REPORT 


1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

• b.  Occasionally  available 

■ 1 c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regularly 

b . Occas i ona 11 y 

c .  Sel  dom  or  never 

4.  How  accurate  is  this  report? 

a.  _____  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  op.nion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d . No  opinion 

6.  The  information  contained  in  this  report  is 

a .  Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 


COMMENTS: 


COURSE/TASK  REPORT  (MONTHLY) 

1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

• b.  Occasionally  available 

' • c.  Seldom  or  never  available' 

3.  How  frequently  do  you  use  this  report? 

a.  Regularly 

b.  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  . Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  opinion  about  the  forrua  t of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d .  No  opinion 

6.  The  information  contained  in  this  report  is 

a .  Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  . Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 

COMMENTS:  _____ 
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COURSE  ABSENTEE  REPORT  (MONTHLY) 

1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

• b. _ Occasionally  available 

' * c.  Seldom  or  never  available 


3.  How  frequently  do  you  use  this  report? 

a . Regul  a r 1 y 

b.  __  Occasionally 

c.  Seldom  or  never 


4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  . Numerous  errors 

d.  Cannot  judge  accuracy  of  report 


5.  What  is  your  opinion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  , Format  is  satisfactory 

c.  Format  requires  revision 

d.  No  opinion 


6.  The  information  contained  in  this  report  is 

a . Essential 

b.  Non-essential 


7.  The  content  of  this  report 


a.  Should  not  be  changed 

b . Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 


8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  margi  nal  use 

c.  Serves  no  useful  purpose 


COMMENTS: 
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COMPANY  ABSENTEE  REPORT  (MONTHLY) 


1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

• b.  Occasionally  available 

' • c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regularly 

b.  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  op.nion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d.  No  opinion 

6.  The  information  contained  in  this  report  is 

a . Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 


COMMENTS:  

% • I 


% 

b * 
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TAIS  REPORT  (MONTHLY) 


1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

■ b.  Occasionally  available  , 

' ' c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regularly 

b .  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  opinion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d . No  opinion 

6 . The  information  contained  in  this  report  is 

a . Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  . Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 

COMMENTS:  
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TEST  ANALYSIS  (MONTHLY) 


1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

• b.  Occasionally  available 

• - c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regularly 

b.  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  ^ Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  opinion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d.  No  opinion 

6.  The  information  contained  in  this  report  is 

a . Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 

COMMENTS:  
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QUESTION  ANALYSIS  (MONTHLY) 

Dlstractor  Count 

1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

b.  Occasionally  available 

' ’ c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regularly 

b.  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  op.nion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d .  No  opinion 

6.  The  information  contained  in  this  report  is 

a.  Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 

COMMENTS:  


quh&iiUN  AimLiaiD  i m. i ) 

Constructed  Response  Printout 


1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

•'/  b.  Occasionally  available 

' • c.  Seldom  or  never  available' 

3.  How  frequently  do  you  use  this  report? 

a . Regul  arl  y 

b . Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a.  _____  Generally  accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  opinion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d.  No  opinion 

6.  The  information  contained  in  this  report  is 

a . Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b.  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a.  Serves  Intended  purpose 

b.  Of  marginal  use 

c.  Serves  no  useful  purpose 


COMMENTS: 
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STUDENT  RECORD  PRINTOUT 
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1.  Are  you  familiar  with  this  report?  Yes  No 

(If  No,  turn  page  to  next  report.) 

2.  Is  this  report  available  to  you? 

a.  Regularly  available 

b.  Occasionally  available 

• - c.  Seldom  or  never  available 

3.  How  frequently  do  you  use  this  report? 

a . Regul  a rl  y 

b.  Occasionally 

c.  Seldom  or  never 

4.  How  accurate  is  this  report? 

a .  General  1 y accurate 

b.  Minor  inaccuracies 

c.  Numerous  errors 

d.  Cannot  judge  accuracy  of  report 

5.  What  is  your  opinion  about  the  format  of  this  report? 

a.  Excellent  format 

b.  Format  is  satisfactory 

c.  Format  requires  revision 

d.  No  opinion 

6.  The  information  contained  in  this  report  is 

a . Essential 

b.  Non-essential 

7.  The  content  of  this  report 

a.  Should  not  be  changed 

b .  Needs  minor  revisions  in  the  content 

c.  Needs  major  revisions  in  the  content 

8.  How  useful  do  you  find  this  report? 

a . 

b. 

c . 


COMMENTS: 


Serves  intended  purpose 
Of  marginal  use 
Serves  no  useful  purpose 
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RESOURCE  ALLOCATION  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 

DATE 

The  purpose  of  this  survey  Is  to  collect  Information  on  the  allocation  of 
resources  within  your  area.  The  results  of  this  will  be  consolidated  with 
other  surveys  to  determine  the  effectiveness  of  the  computerized  training 
system. 

As  you  read  the  statements  and  questions,  make  a mental  comparison  of  your 
training  area  prior  to  and  after  the  imp, I ementatlon  of  the  computerized 
training  system. 

Please  place  a check  mark  by  your  location  and  position  below.  You  do 
not  have  to  sign  the  form. 

Location  Pos i t Ion 

Department  (Opns)  Chief 

Division  NCOIC 

Course  Education  Specialist 

Section  Training  Specialist 

Instructor 

Records  Clerk 


Other 


RESOURCE  ALLOCATION  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
Indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 
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1.  The  computer  system  has  been  able 
to  accurately  account  for  the  lo- 
cation of  each  student  within  the 
course. 


Comments 


2.  The  computer  system  has  main- 
tained an  accurate  accounting  of 
student  position  vacancies. 


3.  When  the  student  completes  a task, 
the  system  routes  him  to  the  next 
task  without  delay. 


4.  Students  are  routed  through  the 
course  according  to  the  predeter- 
mined (normal)  flow. 


5.  Student  routing  has  been  accom- 
plished with  a minimum  of  errors. 


6.  When  student  positions  in  the 
next  sequential  task  are  filled, 
the  student  is  routed  to  an  al- 
ternate task,  for  which  he  has 
the  necessary  prerequisites. 


7.  When  the  student  completes  a task 
and  all  student  positions  in  the 
succeeding  tasks  are  filled,  the 
Instructor  is  alerted  via  the 
hard  copy  terminal . 


RESOURCE  ALLOCATION  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

Strongly  Agree 

Agree 

Nleutra 1 

Di sagree 

Strongly  Disagree 

Comments 

8.  Routing  students  to  the  correct 
student  positions  has  required 
close  monitoring  by  the  instructor/ 
supervisor. 

9.  The  instructor  is  alerted  via  the 
hard  copy  terminal  when  the  student 
has  completed  all  the  required  tasks 

10.  Prior  to  CTS,  did  accounting  for  student  time  in  the  course  place  an 

unnecessary  burden  on  the  classroom  instructor?  Yes No  

Explain. 


11.  Does  the  usefulness  derived  from  using  the  computer  for  accounting  for 
students  time  in  the  CTS  tasks  outweigh  the  workload  Inherent  in  Its 
collection?  Yes No Explain. 
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RESOURCE  ALLOCATION  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


is..  Has  the  computer  system  been  effective  in  accounting  for  student  time 
in  the  course  below  the  task  level?  Yes  No  Explain. 


13.  What  problems,  if  any,  were  encountered  when  a student  was  routed  to 
an  alternate  task  out  of  the  normal  sequence? 


14.  Has  the  computer  system  been  effective  in  directing  student  progress 
through  the  course  via  the  several  learning  alternatives  (modes,  pro- 
gression levels)?  Yes  No  Explain, 


15. 


Do  you  think 
each  student 


It  is  necessary  to  know  the  learning  alternative 
is  working?  Yes  No  Explain. 


In  which 


RESOURCE  ALLOCATION  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


16.  Have  you  had  any  problems  in  determining  the  learning  alternative  in 
which  the  student  is  working?  Yes  No  Explain. 


17.  What  is  your  frank  opinion  of  the  capability  of  the  computer  system 
to  route  and  account  for  students  within  your  course7  (Please  indi- 
cate advantages  and  d isadvantages) 


Advantages 

PI sadvantages 

224 


ATTACHMENT  #8 


I 

I 

I 

I 

i 

I 


r 


I 


INSTRUCTIONAL  PROCFSS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 

Da  t e 

Tho  purpose  of  this  survey  Is  to  gather  Information  concerning  the  com- 
terlzed  instructional  process.  Please  answer  each  item  in  this  survey 
with  complete  candor.  Your  responses  will  be  held  in  strictest  confid- 
ence. If  you  feel  that  you  are  not  In  a position  to  answer  a particular 
question  because  you  have  not  been  closely  associated  with  the  CTS 
project,  please  circle  the  item  number  and  leave  it  blank.  Your  comments 
or  suggestions  will  be  greatly  appreciated. 

Place  a check  (»0  mark  by  your  position  in  your  organization.  You  do 
not  have  to  sign  this  form. 

POSITION 

DIVISION  CHIEF  

SUPERVISOR  TRAINING  INSTRUCTOR  

I NSTRUCTOR 


PLEASE  RECORD  YOUR  ANSWER  OR  CHECK  (^)  THE  ALTERNATIVE  WHICH  BEST 
EXPRESSES  YOUR  REACTION  TO  EACH  ITEM  THAT  FOLLOWS. 
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INSTRUCTIONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
Indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 


SECTION  I.  Optimum  Student  Position/ 
Terminal  Relationship. 

1.  Each  student  carrel  (bench  posi- 
tion) should  be  equipped  with  a 
computer  terminal. 


2.  One  computer  terminal  programmed 
for  every  four  students  has  proved 
to  be  a very  satisfactory  ratio. 


3.  If  each  student  carrel  (bench 

position)  had  a computer  terminal, 
the  utilization  would  not  be  great 
enough  to  make  it  cost  effective. 


Student  queuing  at  the  computer 
terminals  has  not  been  a problem. 


With  approximately  12$  of  the 
course  material  in  the  CAI  mode, 
the  students  have  had  no  diffi- 
culty in  using  the  computer  termi- 
nal whenever  necessary. 


If  the  percentage  of  CAI  is  In- 
creased, it  will  be  necessary  to 
reduce  the  present  4 to  1 student 
to  terminal  ratio  in  the  courses. 


Comments 


INSTRUCTIONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

Strongly  Agree 

Agree 

Neutra 1 

Di sagree 

Strongly  Disagree 

Comments 

7.  Student  to  terminal  ratio  cannot 
be  fixed  at  other  than  one-to-one 
when  operating  primarily  in  the 

CAI  mode. 

8.  Computer  terminals  should  be  in- 
stalled in  the  classrooms  that 
contain  student  carrels  (bench 
positions),  not  in  separate 
c 1 ass rooms . 

g The  only  mode  that  allows  fixing 
an  optional  student  to  terminal 
ratio  is  the  CAI  mode. 

10.  The  terminal  use  time  required  by 
each  student  to  complete  a speci- 
fic unit  will  vary  because  of 
self-pacing. 

11.  To  prevent  aueuinq  problems,  when 
all  student  terminals  are  in  use, 
a provision  should  be  made  for 
the  studpnt  in  the  CMI  mode  to 
obtain  his  next  off-line  assign- 
ment via  the  instructor’s 
(proctor)  terminal. 

1 

12.  Despite  the  erratic  student  flow, 
which  Is  normal  for  self-pacing, 
one  computer  terminal  can  adequat- 
ely handle  four  students. 
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INSTRUCTIONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


J 

Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

13.  Since  many  units  of  instruction 
have  no  CAI,  the  computer  termi- 
nals have  been  Idle  a large 
portion  of  the  time  although 
the  normal  student  to  terminal 
ratio  Is  4 to  1 . 

t 

14.  Clustering  all  the  course  com- 
puter terminals  in  classrooms 
isolated  from  student  carrels 
(bench  positions)  has  enabled 
maximum  utilization  of  the 
terminal s. 

15.  Optimum  student  position/ 

terminal  relationship  can  only 
be  defined  when  the  percentage 
of  CAI  in  the  course  remains 
constant. 

16.  When  the  computer  is  used  for 
both  Instruction  and  student 
management,  no  valid  conclu- 
sions can  be  drawn  with  regard 
to  the  optimum  student  posi- 
tion/terminal ratio. 

. 

17.  A terminal  at  each  student 

carrel  (bench  position)  would 
maximize  student  Interactive 
and  student  managed  instruc- 
tion. 

c a>  l. 
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INSTRUCTIONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


lfl.  Basod  on  student  utilization,  the  optimum  student  position  to  computer 
I ermine  1 ratio  should  be: 

a.  1 to  1 

b.  2 to  1 

c.  3 to  1 

d.  4 to  1 

e.  b to  1 

f.  Other  

19.  Students  would  experience  no  queuing  problems  if  the  student  to  termina 
ratio  did  not  exceed: 

a . 2 to  1 

b.  3 to  1 

c . 4 to  1 

d.  5 to  1 

f.  Other  

20.  An  equipment  laboratory  containing  12  student  carrels  (bench  positions) 
should  have  a minimum  of 

a.  2 terminals  

b.  4 terminals  

c.  6 terminals  

d.  12  terminals  

e.  Other  terminals 
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INSTRUCTIONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Comments  explaining  your 
selection  will  be  appreciated. 

strongly  Agree 

0) 

<d 

O) 

< 

to 

u 

4- 

D 

<D 

7 

Di sagree 

strongly  Disagree 

Comments 

SECTION  II.  Interface  Between  Student 
and  Instructor. 

Note:  A major  problem  area  of  self- 
paced  instruction  has  been  the  limited 
time  the  classroom  instructor  is  avail- 
able to  assist  each  individual  student. 

21.  Since  the  implementation  of  CTS 
the  classroom  Instructor  has  had 
more  time  to  interact  directly 
with  the  student. 

22.  CTS  has  relieved  the  classroom  In- 
structor of  many  of  his  record 
keeping  requirements. 

23.  There  has  been  no  discernible 

change  in  the  availability  of  the 
classroom  instructor  to  Interact 
with  the  student  as  the  result  of 
CTS. 

24.  The  ability  of  the  computer  to 
manage  the  students'  off-line 
activities  has  reduced  the  time 
the  instructor  must  spend  on 
administrative  actions. 

25.  Freeing  the  instructor  of  admini- 
strative duties  is  one  of  the 
most  favorable  aspects  of  CTS. 

INSTRUCT  IONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


SECTION  III.  CTS  Back-up  Capability. 

2b.  An?  adequate  Instructional  materials  available  to  conduct  Instruction 

during  computer  down  time?  Yes  No  If  no,  please 

exp  la i n . 


27.  Is  the  students'  progress  impeded  because  of  the  quality  of  CTS  back-up 

instructional  materials?  Yes  No  If  yes,  please 

explain. 


28.  Have  you  been  able  to  use  printouts  of  onTllne  instructional  materials 
as  back-up  instructional  aids  during  computer  down  time? 

Yes  No  ______  If  no,  please  explain. 
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INSTRUCTIONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


29.  Are  printouts  of  on-line  instructional  materials  available  in  sufficient 

quantities  to  meet  student  needs?  Yes No  If  no,  how  can 

this  be  improved? 


30.  Do  the  classroom  instructors  rely  on  printouts  of  on-line  materials  for 

back-up  instruction  when  the  computer  is  down?  Yes  No  

If  no,  please  explain. 


31.  Is  It  necessary  to  revert  to  instructional  materials  used  prior  to  the 

implementation  of  CTS  for  back-up  during  computer  down  time?  Yes  

No  If  yes,  please  explain. 


32.  Has  computer  down  time  been  more  or  less  of  a problem  than  failure  of 
other  AV  devices  (eg.,  TV,  sound/slides)?  Please  explain. 


233 


INSTRUCTIONAL  PROCESS  SURVEY 
COMPUTERIZED  TRAINING  SYSTEM  (CT3) 


SECTION  IV.  Changes  that  Impact  C;n  Student  Motivation. 

33.  Since  the  Implementation  of  CTS,  have  any  changes  1 hat  would  Impact 
on  student  mol Ivaf Ion  been  nude  in  the  fol lowinn  areas? 


a.  Promotion  (student)  policies 

Yes 

No 

Fxp lain 

1 

1 

b.  Leave  pol icy 

1 

1 

c.  Assignment  policy 

1 

d.  Award  of  MOS 

1 

e.  Major  course  revision 

1 

1 

f . Changes  i n sh  i f ts 
(day  to  night) 

34.  Can  you  identify  any  policy  changes  or  other  actions,  not  cited  above, 
that  may  have  affected  student  motivation  (favorably  or  adversely) 
during  the  CTS  field  test? 
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Attachment  #9 


STUDENT  ATTITUDE  QUESTIONNAIRE  1 

COMPUTERIZED  TRAINING  SYSTEM  (CTS)  INSTRUCTION 

POST  COURSE  DATA  j 

(In  Percentages)  I 

This  is  a questionnaire  to  gather  information  relative  to  computerized  ^ 

training  system  (CTS)  courses  of  instruction.  There  are  no  right  or  ( 

wrong  answers.  Rather,  we  are  interested  in  your  candid  opinion  of  the  j 

following  statements.  Your  complete  frankness  in  recording  your  opin- 
ions will  be  greatly  appreciated.  Individual  responses  will  be  held  in 
strictest  confidence. 

J 

I 


PLEASE  RECORD  YOUR  ANSWER  OR  CHECK  (/)  THE  ALTERNATIVE  WHICH  BEST  EX- 
PRESSES YOUR  REACTION  TO  EACH  ITEM  THAT  FOLLOWS. 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
COMPUTERIZED  TRAINING  SYSTEM  (CTS)  INSTRUCTION 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Explain  your  selection  in 
the  comments,  if  necessary. 

CT>  Q) 

C (L) 

O i- 
t-  cn 
-*->  <C 

CO 

Agree 

Neutral 



Strongly 

Disagree 

— 

SECTION  I.  Course  Content  and  Instruc- 
tional Media. 

1.  The  objectives  of  the  course  are 
clear  and  I know  what  is  expected 
of  me. 

17.65 

38.24 

17.65 

17.65 

8.82 

2.  The  material  in  each  unit  is  orga- 
nized in  a way  that  I can  learn. 

8.82 

23.53 

41.18 

14.71 

11.76 

3.  The  overall  course  content  holds 
my  interest. 

17.65 

32.35 

32.35 

8.82 

8.82 

4.  The  lesson  material  makes  you 
think. 

20.59 

52.84 

23.53 

0.00 

2.94 

5.  I cannot  learn  what  I want  to 
learn  with  this  kind  of  instruc- 
tion. 

8.82 

32.35 

35.29 

11.76 

11.76 

6.  Generally,  the  lessons  are  hard 
to  understand. 

11.76 

38.24 

29.41 

14.71 

5.83 

7.  Generally,  the  lessons  are  too 
long. 

11.76 

26.47 

35.29 

20.59 

5.88 

8.  The  level  of  reading  skill  re- 
quired in  most  lessons  is  too 
high. 

32.35 

32.35 

26.47 

5.88 

2.94 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
COMPUTERIZED  TRAINING  SYSTEM  (CTS)  INSTRUCTION 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Explain  your  selection  in 
the  comments,  if  necessary. 

Strongly 

Agree 

Agree 

Neutral 

Di sagree 

Strongly 

Di sagree 

9.  Performance  examinations  cover 
what  is  presented  in  the  lessons. 

14.71 

32.35 

26.47 

17.65 

8.82 

10.-  Generally,  the  lessons  seem  to  be 
planned  just  for  me. 

8.82 

11.76 

38.24 

26.47 

14.71 

11.  I learn  the  course  material  very 
quickly  using  this  method  of 
instruction. 

8.82 

14.71 

44.12 

23.53 

8.82 

12.  I find  myself  hurrying  through  a 
lesson  to  get  it  over  with  rather 
than  trying  to  learn. 

23.53 

41.18 

20.59 

11.76 

2.94 

13.  I answer  questions  wrong  inten- 
tionally (pretest,  posttest)  in 
order  to  get  more  instruction. 

41.18 

38.24 

17.65 

2.94 

0.00 

14.  I waste  no  time  when  using  this 
method  of  instruction. 

2.94 

20.59 

32.35 

29.41 

14.71 

15.  I do  my  best  as  a result  of  this 
method  of  instruction. 

20.59 

20.59 

26.47 

23.53 

8.82 

16.  I always  know  how  well  I am 
doing  in  this  course. 

5.88 

29.41 

41.18 

8.82 

14.71 

17.  CTS  is  a very  effective  method  of 
instruction. 

8.82 

32.35 

26.47 

14.71 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
COMPUTERIZED  TRAINING  SYSTEM  (CTS)  INSTRUCTION 


Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Explain  your  selection  in 
the  comments,  if  necessary. 

Strongly 

Agree 



<U 

QJ 

S- 

o 

< 

Neutral 

i 

Di sagree 

. J 

Strongly 

Disagree 

1 

18.  I feel  that  no  one  real ly  cares 
whether  1 learn  or  not  using 
this  method  of  instruction. 

14.71 

35.29 

35.29 

14.71 

0.00 

19.  I feel  that  I am  pushed  too 
rapidly  through  the  lesson 
material . 

23.53 

44.12 

20.59 

3.82  ' 

2.94 

20.  An  instructor  is  readily  avail- 
able for  assistance. 

2.94 

20.59 

23.53 

32.35 

20.59 

21.  The  instructors  can  answer  my 
questions. 

20.59 

44.12 

26.47 

5.38 

2.94 

22.  There  is  a good  working  rela- 
tionship between  the  instructors 
and  myself. 

20.59 

29.41 

41.18 

5.83 

2.94 

23.  Background  noise  (voices,  move- 
ment, operation  of  equipment) 
is  distracting. 

14.71 

29.41 

38.24 

8.32 

8.82 

24.  Uorking  in  the  carrels  and 

other  student  positions  becomes 
tiresome  over  a long  period  of 
time. 

14.71 

11.76 

26.47 

35.29 

11.76 

25.  There  are  so  many  devices 

(computer  terminals,  TV/cas- 
settes, slide  projectors)  to 
operate  that  it  distracts  from 
the  instruction. 

17.65 

33.24 

26.47 

11.76 

5.38 
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Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Explain  your  selection  in 
the  comments,  if  necessary. 

Strongly 

Agree 

CD 

<D 

S- 

err 

< 

Neutral 

i 

Disagree 

I 

I Strongly 

Di sagree 

I 

26.  Constant  changes  from  one  in- 
structional media  to  another 
interfers  with  learning. 

11.76 

38.24 

32.35 

8.82 

8.82 

27.  CAI  (student  interacts  with  com- 
puter terminal)  provides  a 
better  learning  environment 
than  other  methods. 

5.88 

14.71 

38.24 

29.41 

11.76 

28.  The  computer  terminal  is  easy 
to  operate. 

32.35 

50.00 

14.71 

2.94 

0.00 

29.  The  computer  terminal  text  dis- 
plays are  clear  and  easy  to  read. 

32.35 

38.24 

23.53 

0.00 

5.88 

30.  The  amount  of  material  presented 
on  the  individual  terminal  dis- 
plays is  not  excessive. 

14,71 

41.18 

35.29 

8.82 

0.00 

31.  The  sequencing  of  the  com- 
puterized lesson  material  (each 
lesson  builds  on  preceding 
lesson)  makes  learning  rela- 
tively easy. 

11.76 

29.41 

41.18 

8.82 

8.82 

32.  The  amount  of  CAI  (student  inter- 
acts with  computer  terminal)  in 
the  course,  compared  to  other 
methods,  is  highly  satisfactory. 

8.32 

17.65 

52.94 

14.71 

5.88 
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Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Explain  your  selection  in 
the  comments,  if  necessary. 

Strongly 

Agree 

Agree 

Neutral 

Di sagree 

Strongly 

Disagree 

33.  The  computer  terminals  are  in- 
operative too  often,  which 
wastes  my  time. 

8.82 

11.76 

33.24 

23.53 

17.65 

34.  If  I had  my  choice,  I would 
prefer  CAI  over  other  methods. 

8.82 

14.71 

29.41 

29.41 

17.65 

35.  Graphic  displays  on  the  com- 
puter terminal  were  sharp  and 
easy  to  understand. 

14.71 

35.29 

41.18 

2.94 

5.88 

36.  I had  no  problem  in  learning  to 
use  the  keyboard  at  the  computer 
terminal . 

35.29 

44.12 

11.76 

2.94 

5.88 

37.  I feel  that  computerized  in- 
struction challenged  me  to  do 
my  very  best. 

26.47 

20.59 

35.29 

3.82 

8.82 

38.  If  I have  the  opportunity  to 
take  another  Army  training 
course,  I would  prefer  CTS  as 
the  method  of  instruction. 

8.82 

26.47 

17.65 

23.53 

23.53 

SECTION  II.  CM I Mode  (Material 
presented  off-line  under  computer 
management). 

39.  Most  of  my  time  in  the  course 
is  spent  in  the  CMI  mode. 

0.00 

38.24 

38.24 

i — i 

14.71 

8.82 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
COMPUTERIZED  TRAINING  SYSTEM  (CTS)  INSTRUCTION 


1 

1 Please  check  the  appropriate  block  to 

I indicate  your  opinion  of  the  following 

{ statements.  Explain  your  selection  in 

the  comments,  if  necessary. 

1 

Strongly 

Agree 

Agree 

Neutral 

r 

Strongly 

Disagree 

40.  I prefer  receiving  all  my  assign- 
ments via  the  computer  terminal. 

5.88 

29.41 

32.35 

23.53 

8.8; 

41.  I feel  that  receiving  my  assign- 
ments via  the  computer  terminal 
has  enabled  me  to  lower  my  pro- 
gression index  (P. I . ) 

8.82 

26.47 

32.35 

14.71 

17.65 

42.  The  instructors  kept  referring  me 
to  the  computer  terminal  for  di- 
rections rather  than  answering 
j my  questions  about  off-line 

assignments. 

8.82 

32.35 

32.35 

23.53 

2.94 

43.  I had  no  difficulty  in  getting 
to  use  a computer  terminal  when- 
ever it  was  necessary. 

5.83 

17.65 

35.29 

23.53 

17. 6E 

44.  I feel  that  the  CTS  instruction 
is  too  impersonal  with  so  much 
time  spent  in  the  CMI  mode. 

5.88 

17.65 

44.12 

29.41 

2.94 

45.  Computer  down  tine  did  not  affect 
' my  progress  through  the  course. 

0.00 

14.71 

23.53 

35.29 

26.47 

46.  It  was  difficult  for  me  to  deter- 
mine my  next  training  task  when 

1 the  computer  system  was  down. 

14.71 

23.53 

35.29 

20.59 

5.38 

► 

| 47.  You  always  know  exactly  where 

1 you  stand  in  the  course  when  in 

the  CMI  mode. 

■ 

20.59 

41.18 

11.76 

17.65 

I 


a 
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Please  check  the  appropriate  block  to 
indicate  your  opinion  of  the  following 
statements.  Explain  your  selection  in 
the  comments,  if  necessary. 

Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 

Disagree 

48.  I like  the  feeling  of  indepen- 
dence associated  with  computer 
managed  instruction. 

14.71 

29.41 

41.18 

11.76 

2.94 

49.  I would  rather  receive  my  off- 
line assignments  from  an  instruc- 
tor so  I can  ask  questions  and 
clarify  any  points  that  I don't 
understand. 

5.88 

20.59 

38.24 

14.71 

20.59 

50.  I would  like  to  have  more  in- 
struction in  the  CAI  mode. 

5.38 

23.53 

47.06 

14.71 

8.82 

51.  I would  rather  go  through  the 
course  at  the  same  speed  as  the 
other  students  as  I seem  to 
learn  more  in  a group. 

17.65 

29.41 

32.35 

11.76 

C\J 

CO 

o 

52.  It  is  easier  to  connect 

learning  elements  in  the  CMI 
mode  than  it  is  under  other 
types  of  instruction  I have 
experienced. 

5.86 

23.53 

4 

50.00 

8.82 

11.76 

53.  I have  experienced  no  problems 
in  moving  through  the  course 
in  either  the  CAI  or  CMI  mode. 

11.76 

20.59 

41.18 

14.71 

11.76 

54.  Instructions  received  via  the 
computer  terminal  are  clear, 
concise  and  easy  to  follow. 

11.76 

44.12 

35.29 

5.88 

2.94 

1 
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1 

Please  check  the  appropriate  block  to 
„ indicate  your  opinion  of  the  following 

, I statements.  Explain  your  selection  in 

the  comments,  if  necessary. 

1 

Strongly 

Agree 

0) 

CL) 

U 

cr> 

C 

Neutral 

Disagree 

Strongly 

Disagree 

i 55.  I feel  that  my  progression  index 

1 (P.I.)  would  be  lower  if  my  course 

activities  were  controlled  by  the 
classroom  instructor. 

11.76 

11.76 

41.18 

26.47 

8.82 

56.  Not  being  able  to  ask  the  instruc- 
tor a question  immediately  when  a 
problem  occurred  has  made  learning 
more  difficult  for  me. 

5.88 

14.71 

44.12 

20.59 

14.71 

57.  There  were  not  enough  computer 
terminals  in  the  course  to  meet 
'1  all  the  student  requirements. 

20.59 

35.29 

29.41 

5.88 

8.82 

I 58.  Too  much  waiting  occurred  because 

the  computer  terminals  were  down 
for  maintenance. 

I 

14.71 

29.41 

41.18 

5.88 

8.82 

I 

59.  Waiting  to  get  to  a computer 
• terminal  caused  me  to  lose  time 

| in  getting  through  the  course. 

14.71 

26.47 

38.24 

14.71 

5.88 

| 60.  I would  prefer  to  have  a computer 

, ■ terminal  at  each  classroom  posi- 

tion. 

■ 

23.53 

32.35 

29.41 

8.82 

5.88 

1 

61 . In  my  course  there  were  too  many 
_ students  for  the  number  of  com- 

| put er  terminals  available. 

» 

5.88 

5.88 

26.47 

35.29 

26.47 

| 62.  The  best  configuration  for  CTS 

• is  to  cluster  the  computer  termi- 

nals in  one  classroom. 

1 

2.94 

5.88 

38.24 

29.41 

23.53 
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COMPUTERIZED  ERA  IN  INC  SYSTEM  (CTS)  INSTRUCTION 


SECTION  III.  On-Line  Time  per  Session. 

63.  Did  you  have  any  experience  working  with  computer  terminals  in  a class- 
room prior  to  taking  this  course? 

Yes  23. 53  n0  76.47 

64.  Did  you  complete  a lesson  unit  of  2 hours  or  more  using  a computer 
terminal  as  the  instructional  medium  prior  to  this  course? 

Yes  29.41  No  70.59 

65.  Did  you  receive  any  instruction  on  the  operation  of  computer  terminals 
prior  to  entering  this  course? 

Yes  38. 24  No  61 . 76 

66.  Did  you  have  any  difficulty  in  learning  to  operate  the  computer  terminal? 

Yes  U.76  No  88.24 

t 67.  How  long  did  it  take  you  to  learn  to  operate  the  computer  terminal  in 

this  course? 

a . 70.59  15  min. 

b . 17.65  30  min. 

c . 2. 94  45  m j n , 

d.  60  min. 

e.  ^ *94  Q-fhop  (Show  time) 

68.  What  was  the  average  time  you  spent  interacting  with  the  computer  ter- 
minal during  a single  session  in  the  CAI  mode? 

a.  2.94  10  min. 

b.  23.53  20  min. 

c.  35.29  30  min. 

d.  14.71  40  min. 

e.  ^.82  50  min. 

f.  1^-71  Other  (show  lime) 
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STUDENT  ATTITUDE  QUESTIONNAIRE 
COMPUTERIZED  TRAINING  SYSTEM  ( CT5 ) INSTRUCTION 


69.  At  what  point  in  time  did  you  begin  to  feel  uncomfortable  working  at 
the  computer  terminal  wilhout  a break? 


a . 

0.00 

After 

10 

mi  nutes 

b. 

23.53 

After 

20 

mi  nutes 

c. 

29.41 

After 

30 

mi  nutes 

d. 

8.32 

After 

40 

minutes 

e. 

14.71 

After 

50 

mi  nutes 

f . 

23.53 

Other 

(show  time) 

70.  What  is  the  maximum  period  (except  for  normal  breaks)  you  spent  in 
direct  contact  with  the  terminal  during  any  one  training  day  that  you  worked 
with  the  computer? 


a<  32,35 

b.  26.47 

c.  26.47 

d.  J^8Z 

„ 5.88 


1 hour 

2 hours 

3 hours 

4 hours 

Other  (show  time) 


7).  Based  on  your  experience,  what  is  the  maximum  time  a student  should 
spend  at  the  computer  terminal  without  a break? 


a.  26.47 

b. 

c.  iHL 

d.  5-88 

0.00 


30  mi nutes 
45  minutes 
60  minutes 
75  minutes 
Other  (show  time) 
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STUDENT  ATi  !TUDE  QUESTIONNAIRE 
COMPUTERIZED  TRAINING  SYSTEM  (CIS)  INSTRUCTION 


72.  If  you  could  establish  tho  time- length  per  session,  how  much  time  would 
you  schedule  a student  to  spend  at  the  computer  terminal? 

a.  Withouf  a break  - 

( 1 ) 5.88  10  mi nutes 


(2) 

17.65 

20  minutes 

(3) 

23.53 

30  minutes 

(4) 

23.53 

40  mi nutes 

(5) 

23.53 

50  minutes 

(6) 

5.88 

Other  (show  time) 

b. 

With  a 

10  minute  break  between  sessions 

(1) 

17.65 

30  minutes 

(2) 

47.06 

1 hou  r 

(3) 

26.47 

1 i hours 

(4) 

5.88 

2 hours 

(5) 

0.00 

3 hours 

(6) 

2.94 

Other  (show  time) 

73.  In  your  opinion,  the  ideal  time  length  for  interaction  with  the  com- 
puter lerminal  would  be  how  many  minutes  out  of  each  50  minute  period? 


a. 

11.76 

10 

mi nutes 

b. 

35.29 

15 

mi  nutes 

c. 

8.82 

20 

mi  nutes 

d. 

14.71 

25 

mi  nutes 

0. 

11.76 

30 

mi nutes 

f. 

17.65 

01  her 
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4 


74.  In  your  opinion,  the  ideal  time-length  for  interaction  with  the  com- 
puter terminal  r.hould  not  exceed  how  many  hours  in  any  one  day? 

a.  fi. n?  1 hour 

b.  26.47  2 hours 

c.  41.18  3 hours  , 

d.  17.65  ^ hours 

e . 5.88  Other 

SECTION  IV.  Acceptance  of  Compel ei  .ed  Instruction  \ 

75.  What  do  you  like  best  about  computerized  instruction  (rank  in  order  of 
pref erence) ? 

a.  ^ I can  0°  a^  mY  own  speed. 

b.  20.59  It  presents  materials  in  a clear  and  interesting  way. 

e.  18.82  I am  always  being  asked  questions.  . 

d.  18,24  I like  the  freedom  offered  by  CTS. 

e.  17.65  I am  not  bothered  by  an  instructor  except  when  I need  him. 

f.  0.00  Other  (specify)  


76.  What  do  you  like  least  about  computerized  instruction  (rank  in  order 
of  preference)? 

a.  22.35  I cannot  ask  questions. 

b.  18,?4  It  is  too  much  work. 

c.  1 9. 41  I have  to  learn  to  operate  too  much  instructional  equipment. 

d.  18.82  It  is  too  impersonal. 

e.  21.18  It  leaves  out  too  much  information  that  an  instructor  would 

provide. 

g.  A' 00  Of  her  (specify) 
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STUDENT  ATTITUDE  QULSTIONNA 
COMPUTERIZED  TRAINING  SYSTEM  (CTS) 


RE 

NSTRUCT I ON 


77.  Which  lessons  or  part  of  the  course  caused  you  the  most  difficulty? 
(Be  as  specific  as  possible.) 


78.  Which  instructional  devices  and  classroom  (working)  conditions  cause 
the  most  annoyance?  (Be  as  specific  as  possible.) 

a.  Instructional  devices  - 


b.  Classroom  (working)  conditions  - 


79.  What  is  your  frank  opinion  concerning  the  relative  worth  of  computerized 
instruction  versus  other  methods  of  instruction  with  which  you  are  familiar? 
(Please  identify  other  methods  of  instruction.) 


NOTE:  Please  conpleto  Iho  items  on  the  last  page,  detach  and  return  to  the 
instructor. 
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- DETACH  - 

Background  Information 


To  insure  that  all  students  complete  Iho  questionnaire,  you  are  asked  to 
answer  the  following  items,  defach  from  the  questionnaire,  and  return  to 
the  instructor. 


DATE: 


NAME  : 


COURSE : 


Check  the  approximate  time  you  have  been  in  this  course: 

a.  Three  weeks . 

b.  Completed  the  entire  course . 


- DETACH  - 
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ATTACHMENT  //1(J 


. MEMORANDUM  OF  UNDERSTANDING 
BETWEEN 


Product  Manager  Commandant 

Computerized  Training  System  US  Army  Signal  School 

Fort  Monmouth,  New  Jersey  Fort  Gordon,  Georgia 

1.  INTRODUCTION: 

a.  References: 

(1)  USASESS  Reg  10-2,  subject:  Organization,  Mission  and  Functions, 
dated  July  1973. 

(2)  Prototype  Computerized  Training  System  Management  Plan  (as 
revised  13  Nov  73). 

(3)  Product  Manager  Charter,  Prototype  Computerized  Training 
System  (as  revised  5 Nov  73). 

(4)  Letter,  ATSN-CTS,  USASCS,  29  Nov  73,  subject:  Designation 

of  Courses  for  Prototype  Computerized  Training  System  (CTS),  with  1st  Ind 
ATTS-ITR,  TRADOC,  11  Dec  73. 

b.  In  accordance  with  cited  references,  this  Memorandum  of  Under- 
standing delineates  responsibilities,  command  and  control  channels,  and 
procedures  to  be  followed  in  the  operational  test  and  evaluation  of  a 
Prototype  Computerized  Training  System  (Project  ABACUS)  by  the  Office 
of  the  Product  Manager,  Fort  Monmouth,  and  the  US  Army  Signal  School 
(USASIGS),  Fort  Gordon,  Georgia. 

2.  GENERAL  RESPONSIBILITIES: 

a.  The  Commandant,  USASIGS,  reference  la  (1),  is  responsible  for 
preparing,  conducting  and  administering  course  of  instruction/programs 
of  instruction  (POI). 

b.  The  Commandant,  USASIGS,  reference  la  (2),  is  responsible  for 
monitoring  and  supporting  the  three  courses  reference  la  (4)  involved  in 
the  operational  phase  of  Project  ABACUS  for  the  Product  Manager. 


c.  The  Product  Manager,  CTS,  reference  la  (3),  is  responsible  for 
the  design,  hardware /software  development,  course  development,  opera- 
tion, and  evaluation  of  Project  ABACUS.  In  this  capacity  the  Product 
Manager,  CTS,  will  advise  the  Commandant,  USASIGS,  during  the  opera- 
tional test  period  in  those  areas  of  responsibility  as  they  apply  to  the 
operational  test  and  evaluation. 

3.  DETAILED  RESPONSIBILITIES: 

a.  The  Product  Manager,  CTS,  will: 

(1)  P rocure,  deliver,  install  and  conduct  the  acceptance  test  of  the 
hardware/software  system  for  operation  at  USASIGS,  Fort  Gordon. 

(2)  Provide  necessary  interfacing  between  USASIGS  and  other  activi-. 
ties,  elements,  and  contractors  in  matters  pertaining  to  the  prototype 
operational  test  and  evaluation. 

(3)  Develop  the  evaluation  plan;  designate  data  to  be  recorded  and 
required  format;  collect  and  analyze  test  data  and  prepare  the  formative, 
interim,  and  final  evaluation  reports. 

(4)  Prepare  appropriate  program  and  planning  documents  as  required 
by  and  for  submission  to  higher  headquarters.  Documents  applicable  to 

the  operational  test  and  evaluation  will  be  coordinated  with  the  Commandant, 
USASIGS.  The  final  document  shall  include  Commandant,  USASIGS 
concur  rence/ comments. 

(5)  E stablish  and  provide  to  the  Commandant,  USASIGS,  operational 
test  facilities  requirements. 

(6)  Provide  a training  program  for  USASIGS  system  and  course  person- 
nel in  the  system,  the  ins  true  tic  cal  model,  and  course  development 
techniques . 

(7)  Establish  and  maintain  a field  office  at  USASIGS,  Fort  Gordon, 
with  the  missions  and  functions  and  phased  personnel  augmentation  as  set 
forth  at  Inclosure  1.  The  organizational  relationships  between  the  field 
office  and  the  USASIGS  task  group  are  as  indicated  at  Inclosure  3. 
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b.  The  Commandant,  USASIGS,  will: 

(L)  Provide  appropriate  facilities  for  the  operational  test  in  accord- 
ance with  requirements  established  by  the  Product  Manager. 

(2)  Coordinate  with  appropriate  Fort  Gordon  activities  to  support  the 
installation  of  the  hardware /software  system  at  Fort  Gordon,  as  required 
by  the  Product  Manager, 

(3)  Provide  administrative  support  for  the  acceptance  testing  at  Fort 
Gordon,  as  required  by  the  Product  Manager. 

(4)  Operate  and  maintain  the  hardwarc/software  system  after  the  final 
contract  acceptance  of  the  system  by  the  Product  Manager  until  the  con- 
clusion of  the  operational  test  and  evaluation  with  the  monitorship  and 
advice  of  the  Product  Manager. 

(5)  Plan,  program  and  budget  additional  system  capacity  not  required 
by  the  operational  test  and  evaluation.  Any  additional  use  of  the  system 

in  this  manner  will  be  limited  to  Project  ABACUS  applications  to  be  jointly 
concurred  in.  The  Product  Manager  CTS  will  retain  approval  authority  for 
recommended  system  use  that  is  in  addition  to  the  test  and  evaluation. 

(6)  Accept  full  responsibility  for  the  system  established  at  Fort  Gordon 
at  the  conclusion  of  the  operational  test  and  evaluation  for  use  in  accordance 
with  TRADOC  r&gulations  in  effect  at  that  time. 

(7)  Provide  the  USASIGS  Task  Group  leader  and  implementation  staff 
as  indicated  at  Inclosure  2 to  conduct  the  course  development  and  opera- 
tional test  of  the  prototype  system. 

(8)  Prepare,  conduct  and  administer  the  POI  for  the  operational  test 
and  evaluation  with  the  advice  and  monitorship  of  the  Product  Manager. 

(9)  Record  evaluation  data  under  specified  training/ testing  conditions 
in  the  required  format  as  designated  by  the  Product  Manager. 

(10)  Provide  administrative  working  space  for  the  PMO  field  liaison 
office  and  coordinate  norma]  post  support. 


4.  OTHER: 


a.  Funding:  Product  Manager,  CTS,  is  responsible  for  program- 
ming and  funding  for  FY  1975  and  1976  as  pertains  to:  hardware  and 
computer  software  acquisition  and  associated  contract  change  orders, 
systems  maintenance,  interconnecting  video  and  signal  cables  between 
the  display  controller  and  student  terminals.  The  Commandant,  USAS1GS, 
will  provide  FY  75/76  O&MA  funds  for  normal  operating  supplies,  com- 
puter consumables,  and  magnetic  devices  (i.c.,  magnetic  tape,  discs, 
and  disc  packs).  The  Commandant  will  also  provide  for  system  operations 
and  maintenance  effective  1 July  1976. 

b.  Responsibilities  for  items  not  s ecifically  covered  in  tliis  Memo- 
randum of  Understanding  will  be  resolved  by  mutual  coordination. 

c.  This  Memorandum  of  Understanding  is  subject  to  review  and 
revision  on  the  anniversary  date  of  the  Product  Manager  Charter  or  when 
major  program  changes  are  made  by  higher  headquarters. 

d.  This  Memorandum  of  Understanding  is  effective  on  date  of  signing 
by  the  Product  Manager,  CTS  and  Commandant,  USASIGS. 


G.  B.  HOWARD,  COL 
PRODUCT  MANAGER,  CTS 


CHARLES  R.  MYER,  MG 
COMMANDANT,  USASIGS 


DATE  4 December  1974 


DATE  13  January  1975 


3 Incl 
as 
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